Interface control identification detection model method based on improved YOLOv3
By improving the YOLOv3 model, using ResNet50_vd backbone network, variable convolution DCN, SE attention mechanism and PANet structure, the problem of insufficient detection speed and accuracy of YOLOv3 is solved, and more efficient software automated test control recognition is achieved.
Patent Information
- Application Number
- CN202411150695.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-26
AI Technical Summary
The existing YOLOv3 algorithm has a low detection speed in software automation testing, and the detection accuracy cannot meet the high coverage requirements of modern software automation testing.
By replacing the backbone network of the YOLOv3 model with ResNet50_vd, and introducing variable convolution DCN and SE attention mechanisms, modifying the model neck network to PANet, adding SPP space pooling pyramid and DropBlock regularization methods, performing image preprocessing and feature Extract and identify.
It improves model detection speed and detection accuracy, meets the high coverage requirements of modern software automation testing, and is especially suitable for control recognition in the software interface of the oil and gas industry.
Smart Images

Figure CN119027784A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of software automated testing interface control recognition, and in particular to an interface control recognition detection model method based on improved YOLOv3. Background Art
[0002] Vigorously promote the deep integration of informatization and industrialization, promote the transformation and innovation of the industry in all directions, all angles and the entire chain with the new generation of information technology, and stimulate the amplification, superposition and multiplication of data on economic development. Software automation testing, as an important automation testing method for the digitalization and intelligence of Industry 4.0, uses an automation testing tool to verify the needs of various software tests, including the management and implementation of testing activities, and the development and execution of test scripts. Intelligent software automatic testing is conducive to the unified establishment of industry customized software development standards, greatly improving testing efficiency, ensuring software security, and improving software quality, empowering the digital transformation and intelligent development of the industrial Internet.
[0003] Software automation testing for the oil and gas industry participates in the entire process of data collection and transmission to value generation in the oil and gas industry chain. It is an essential carrier for the digital transformation of the oil and gas industry and a core component of the oil and gas industrial Internet. At present, domestic software testing is still dominated by manual testing, with low reliability and work efficiency and difficulty in achieving full coverage. China urgently needs an efficient and intelligent software testing tool platform. Intelligent software automatic testing is conducive to the unified establishment of industry customized software development standards, greatly improving testing efficiency, ensuring software security, and improving software quality, empowering the digital transformation and intelligent development of the oil and gas industry.
[0004] YOLOv3 is the third version of the YOLO series algorithm. It uses Darknet-53 as the backbone network, further deepens the network depth, and improves the backbone feature extraction capability. Compared with the previous YOLOv2 backbone network, the speed is reduced, but the accuracy is greatly improved. If the current YOLOv3 algorithm is used for model detection, the speed is low and the detection accuracy needs to be improved, which cannot meet the high coverage requirements of modern software automated testing. Summary of the invention
[0005] The technical problem to be solved by the present invention is: in order to solve the problems existing in the prior art in the above-mentioned background technology, a method for interface control recognition and detection model based on improved YOLOv3 is provided.
[0006] The technical solution adopted by the present invention to solve the technical problem is: an interface control recognition detection model method based on improved YOLOv3, comprising the following steps: S1. Take a screenshot of the software interface, and mark the control UI that needs to be identified in the screenshot to form a data set, pre-process the data set and mark it as a neural network training sample; S2. Replace the backbone network of the YOLOv3 model with ResNet50_vd, introduce variable convolution DCN, and add SE attention mechanism after each block and before shortcut in the bottleneck part of the backbone network; S3. Modify the model neck network to PANet, add SPP spatial pooling pyramid, and introduce DropBlock regularization method; S4. Use the processed data set to train the improved model, which includes three steps: image preprocessing, feature extraction, and image recognition.
[0007] Furthermore, the annotation in step S1 includes adding labels, classifications or annotations to the data, and the preprocessing of the data set includes data cleaning, data preprocessing, data annotation, establishing annotation specifications, data review and correction, and dividing the training, verification and test sets.
[0008] Furthermore, the data cleaning includes removing defective screenshots and low-resolution screenshots, the data preprocessing includes segmentation or re-screenshotting, and the data annotation includes labeling or annotating the control UI in the screenshot.
[0009] Furthermore, in the ResNet50_vd neural network of step S2, the downsampling is moved to the 3×3 convolution, the downsampling in the identity mapping is completed using average pooling, and the variable convolution DCN adds an offset to the sampling position in the standard convolution operation. The offset is obtained by replacing the backbone network 3×3 convolution with DCN, and the deviation is output after the input feature map.
[0010] Furthermore, the SE attention mechanism in step S2 learns the correlation between channels through SENet, filters out the attention for the channels, and assigns different weights to different positions of the image from the perspective of the channel domain through the weight matrix, thereby obtaining more important feature information.
[0011] Furthermore, in step S3, an SPP spatial pooling pyramid is added to the PANet network, and the feature layers are respectively passed through a maximum pooling layer with a pooling kernel size of 5×5, 9×9, and 13×13, and then spliced in the channel direction for further fusion. The DropBlock regularization method randomly masks a part of the continuous area and removes certain semantic information to play a regularization role, thereby controlling the number of masked features and satisfying the Bernoulli distribution.
[0012] Furthermore, the image preprocessing in step S4 is to preprocess the image to be identified through operations including grayscale, binarization, denoising, smoothing, etc., so as to remove interference and noise and enhance the feature information of the image; feature extraction is to extract effective features from the preprocessed image through SIFT, HOG or SURF methods for use by subsequent classifiers; image recognition is to identify objects in the image through the extracted features through deep learning methods.
[0013] Beneficial effects of the present invention: The present invention introduces the ResNet50_vd neural network, adds a variable convolution DCN to the neural network to expand the training range; improves the FPN structure to the PANet structure to realize bidirectional feature transfer, adds the SPP spatial pooling pyramid to the PANet, improves the image scale invariance while reducing overfitting, and accelerates the convergence of the training process; at the same time, introduces the DropBlock regularization method to effectively remove certain semantic information to play a regularization role; also adds the SE attention mechanism to filter out the attention for the channel, and assigns different weights to different positions of the image from the perspective of the channel domain through the weight matrix, so as to obtain more important feature information.
[0014] The present invention improves the backbone network and neck network of the recognition and detection model, introduces variable convolution, spatial pooling pyramid and attention mechanism respectively, improves the model detection speed and detection accuracy, meets the high coverage requirements of modern software automated testing, and is particularly suitable for control recognition in software interfaces in the oil and gas industry. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0016] Figure 1 It is a flow chart of the present invention.
[0017] Figure 2 It is a flow chart of forming the neural network training samples in the present invention.
[0018] Figure 3 It is a schematic diagram of the basic model of the backbone network of the present invention.
[0019] Figure 4 Schematic diagram of the basic model of the attention mechanism of the present invention.
[0020] Figure 5 Schematic diagram of the basic model of the spatial pooling pyramid of the present invention. DETAILED DESCRIPTION
[0021] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.
[0022] like Figure 1 As shown, an interface control recognition detection model method based on improved YOLOv3 includes the following steps: S1. Take a screenshot of the software interface, and mark the control UI that needs to be identified in the screenshot to form a data set, pre-process the data set and mark it as a neural network training sample; S2. Replace the backbone network of the YOLOv3 model with ResNet50_vd, introduce variable convolution DCN, and add SE attention mechanism after each block and before shortcut in the bottleneck part of the backbone network; S3. Modify the model neck network to PANet, add SPP spatial pooling pyramid, and introduce DropBlock regularization method; S4. Use the processed data set to train the improved model, which includes three steps: image preprocessing, feature extraction, and image recognition.
[0023] Among them, the data set annotation in step S1 is the process of processing the original data to make it meet the needs of deep learning. In view of the characteristics of the oil and gas industry software interface in this embodiment, such as the large number of control icons and small targets, it is usually necessary to manually take screenshots of the software interface and manually annotate the control UI that needs to be identified in the screenshot. Annotation usually includes operations such as adding labels, classifications or annotations to the data so that the data can be correctly identified and understood when training the model. The annotated data set is used to train, verify and test the improved model. Data set annotation includes data cleaning, data preprocessing, data annotation, establishing annotation specifications, data review and correction, dividing training, verification and test sets, etc. Steps such as Figure 2 As shown in the figure, before starting to label, the existing screenshots of the oil and gas industry software interface should be screened and processed to remove defective screenshots and low-resolution screenshots; the cleaned screenshots of the oil and gas industry software interface should be properly preprocessed, such as segmentation, re-screenshot and other operations to better adapt to the deep learning algorithm; according to the required recognition task, the control UI in the preprocessed screenshots of the oil and gas industry software interface should be manually labeled or annotated; in order to ensure the consistency and accuracy of the labeling, it is necessary to establish clear labeling specifications to label the control UI in the screenshots of the oil and gas industry software interface; after the labeling is completed, the control UI labels in the screenshots of the oil and gas industry software interface need to be reviewed and corrected; finally, the labeled screenshots of the oil and gas industry software interface are divided into training, validation and test sets as needed. The training set is used to train the model, the validation set is used to adjust the model parameters and select the best model, and the test set is used to evaluate the generalization ability of the model.
[0024] The downsampling operation in the original model network backbone ResNet is implemented in the residual structure using a 1×1 convolution with a step size of 2. The input data will be reduced to half the size of the feature map through a 1×1 convolution with a step size of 2. The 1×1 convolution with a step size of 2 will cause the input feature map to Figure 3 / 4 information is not used. Therefore, this embodiment uses the network backbone structure of ResNet50_vd, such as Figure 3 As shown in the figure, the downsampling is moved to the 3×3 convolution, and the downsampling in the identity map is completed using average pooling. The variable convolution, i.e. DCN, adds an offset to the sampling position in the standard convolution operation. The offset is obtained by replacing the backbone network 3×3 convolution with DCN, and the deviation is output after the input feature map.
[0025] In step S2, the SE attention mechanism is introduced. SENet learns the correlation between channels, filters out the attention for the channels, and assigns different weights to different positions of the image from the perspective of the channel domain through the weight matrix, thereby obtaining more important feature information. The algorithm flowchart can be divided into four steps: F tr Transformation transforms feature map X into U, and global information embedding will contain global information W×H×C The feature map of is directly compressed into a 1×1×C feature vector. The channel features of C feature maps are compressed into a value, so that the generated channel-level statistics contain contextual information; adaptive recalibration uses the information gathered in the compression operation and adopts a gate mechanism consisting of two layers of full connection: the first fully connected layer compresses C channels into C / r channels to reduce the amount of calculation, and the second fully connected layer restores the number of channels to C channels, and then obtains the weight through Sigmoid activation; reweighting weights the attention weights obtained previously to the features of each channel, and multiplies each feature map in the feature map U by the corresponding weight to obtain the final output of SENet . Add SEblock after each block of the Bottleneck module and before the shortcut, so that SEblock is embedded in the residual module of ResNet, as shown in Figure 4 shown.
[0026] In step S3, a bottom-up pyramid is added as a supplement on the basis of FPN. By transferring the bottom-level positioning features to the top, the semantic information is combined with the positioning information, thus realizing the bidirectional feature transfer. The SPP spatial pooling pyramid is added to the PANet network, such as Figure 5As shown in the figure, the feature layers are passed through a maximum pooling layer with a pooling kernel size of 5×5, 9×9, and 13×13, and then spliced in the channel direction for further fusion. The DropBlock method randomly masks a part of the continuous area, which can effectively remove some semantic information and thus play a regularization role. γ is the probability in the drop process, which controls the number of masked features and satisfies the Bernoulli distribution (i.e., discrete probability distribution).
[0027] In step S4, image preprocessing is to preprocess the image to be identified through operations including grayscale, binarization, denoising, smoothing, etc., so as to remove interference and noise and enhance the feature information of the image; feature extraction is to extract effective features from the preprocessed image through methods such as SIFT, HOG, SURF, etc. for use by subsequent classifiers; image recognition is to identify objects in the image through the extracted features through deep learning methods.
[0028] In summary, this embodiment improves the backbone network and neck network of the recognition and detection model by introducing variable convolution, spatial pooling pyramid and attention mechanism, respectively, so as to improve the model detection speed and detection accuracy and meet the high coverage requirements of modern software automated testing.
[0029] Specifically: the ResNet50_vd neural network is introduced, variable convolution DCN is added to the neural network to expand the training range, and the FPN structure is improved to the PANet structure to realize bidirectional feature transfer, and the SPP spatial pooling pyramid is added to PANet to improve the image scale invariance while reducing overfitting and accelerating the convergence of the training process; at the same time, the DropBlock regularization method is introduced to effectively remove certain semantic information to play a regularization role; the SE attention mechanism is added to filter out the attention for the channel, and different weights are given to different positions of the image from the perspective of the channel domain through the weight matrix, so as to obtain more important feature information.
[0030] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.
Claims
1. A method for identifying and detecting an interface control model based on improved YOLOv3, characterized in that: The following steps are involved: S1. Take a screenshot of the software interface, and mark the control UI that needs to be identified in the screenshot to form a data set, pre-process the data set and mark it as a neural network training sample; S2. Replace the backbone network of the YOLOv3 model with ResNet50_vd, introduce variable convolution DCN, and add SE attention mechanism after each block and before shortcut in the bottleneck part of the backbone network; S3. Modify the model neck network to PANet, add SPP spatial pooling pyramid, and introduce DropBlock regularization method; S4. Use the processed data set to train the improved model, which includes three steps: image preprocessing, feature extraction, and image recognition.
2. The interface control recognition detection model method based on improved YOLOv3 according to claim 1, characterized in that: The labeling in step S1 includes adding labels, classifications or annotations to the data, and the preprocessing of the data set includes data cleaning, data preprocessing, data labeling, establishing labeling specifications, data review and correction, and dividing the training, verification and test sets.
3. The interface control recognition detection model method based on improved YOLOv3 according to claim 2 is characterized in that: The data cleaning includes removing defective screenshots and low-resolution screenshots, the data preprocessing includes segmentation or re-screenshotting, and the data annotation includes labeling or annotating the control UI in the screenshot.
4. The interface control recognition detection model method based on improved YOLOv3 according to claim 1, characterized in that: In the ResNet50_vd neural network of step S2, the downsampling is moved to the 3×3 convolution, the downsampling in the identity mapping is completed using average pooling, and the variable convolution DCN adds an offset to the sampling position in the standard convolution operation. The offset is obtained by replacing the backbone network 3×3 convolution with DCN, and the deviation is output through the input feature map.
5. The interface control recognition detection model method based on improved YOLOv3 according to claim 1, characterized in that: The SE attention mechanism in step S2 is to learn the correlation between channels through SENet, filter out the attention for the channels, and assign different weights to different positions of the image from the perspective of the channel domain through the weight matrix, so as to obtain more important feature information.
6. The interface control recognition detection model method based on improved YOLOv3 according to claim 1, characterized in that: In step S3, the SPP spatial pooling pyramid is added to the PANet network, and the feature layers are respectively passed through a maximum pooling layer with a pooling kernel size of 5×5, 9×9, and 13×13, and then spliced in the channel direction for further fusion. The DropBlock regularization method randomly masks a part of the continuous area and removes certain semantic information to play a regularization role, thereby controlling the number of masked features and satisfying the Bernoulli distribution.
7. The interface control recognition detection model method based on improved YOLOv3 according to claim 1, characterized in that: The image preprocessing in step S4 is to preprocess the image to be identified by operations including grayscale, binarization, denoising, smoothing, etc., so as to remove interference and noise and enhance the feature information of the image; Feature extraction is to extract effective features from the preprocessed image through SIFT, HOG or SURF methods for use by subsequent classifiers; image recognition is to identify objects in the image based on the extracted features through deep learning methods.
Citation Information
Cited By
Navigation logistics real-time dangerous goods identification system and method
CN120564137A
