A software recognition method based on screen monitoring

Abstract

The present invention discloses an application program recognition method based on screen monitoring, which is improved compared with the traditional video image recognition method, and includes the following steps: Step S1: Use the Leaders clustering algorithm to preprocess the data set, then perform a sampling operation, extract several sample subsets, perform kernel K-means clustering operation on the sample subsets, and finally determine the initial position of the anchor by calculating the mean distance between each sample subset to integrate the clustering results and send it into the YOLOv3 model for object detection; Step S2: Introduce the inception structure that can be used to build a sparse network, so that the network layer with complete parameter connections becomes a sparse network layer, while reducing the number of training parameters and widening the width of the network; Step S3: Improve the multi-scale fusion operation of the YOLOv3 model, consider the target information of the shallower layer of the feature extraction network, and improve the sensitivity of the network to small targets.

Description

Technical Field

[0001] The present invention is a software recognition method based on screen monitoring, belonging to the technical field of application program system recognition. Background Art

[0002] The behaviors of application programs are mainly manifested in human-computer interaction behaviors (interfaces). Therefore, the collection of operation information of programs can be specifically carried out to the collection of interface information. In some large application systems, due to a large number of programs, non-standardized interfaces such as no title, dynamically created spaces, and dynamically changed control IDs, it will cause problems such as low collection efficiency and difficulty in identifying some interfaces, and cannot meet the actual needs. Therefore, this method adopts a method combined with image processing technology to detect screen application programs. Image recognition and processing technology is a new technology that emerged with the rise of the computer industry in the 1960s and has obtained important application results in actual applications and developments. As the saying goes, the most important sense is vision, and images are the basis of vision. The goal of early image processing was to simply optimize the quality, taking people as objects to achieve the purpose of optimizing the visual effect. The principle of image recognition technology is to find similar data prototypes in its own data storage library according to the information recognition results of the technology, and then identify the images.

[0003] In the present invention, a convolutional neural network is used for analysis and prediction processing. At present, deep learning technology has made great progress in the field of image recognition, and various applications emerge in an endless stream, such as face recognition, license plate recognition, medical image recognition and other different industries. Face recognition technology is an image recognition technology that determines the identity of a person based on facial feature information and has been applied in various fields. Face recognition technology based on deep learning is a research hotspot in face recognition and provides impetus for the development of the face recognition field. In 2014, Taigman Y et al. constructed the DeepFace network model, which consists of an 8-layer network structure, including 5 convolutional layers, 1 pooling layer and 2 fully connected layers. This model was trained on 4 million face images and predicted on the LFW dataset with an accuracy of 97.35%. The LFW dataset is a commonly used face image dataset at present, containing 13,233 images and 5,749 people. In 2015, Zhou E et al. constructed the Face++ network model, which is a CNN network containing 10 convolutional layers. At the same time, Zhou E et al. constructed a dataset containing 5 million face images for training and tested it on the LFW dataset with an accuracy of 99.5%. Sun Y et al. constructed the DeepID model, which improved the VGG-Net structure and the GoogLeNet structure respectively, and the accuracy of both structures in the LWF dataset was higher than 97.45%. Schroff F et al. proposed the FaceNet model, which is based on the CNN network, maps face images to Euclidean space feature vectors, and judges the similarity of face images according to the spatial distance. At the same time, this model was trained on face data containing 8 million people, and the accuracy on the LFW dataset was 99.63%. Liu J et al. proposed the Baidu model, which uses multiple GPUs to simultaneously recognize the segmented facial images, and each network contains 9 convolutional layers. The accuracy of this model in the LFW dataset is 99.77%. At present, many scholars are studying face recognition problems. In 2019, Hu Yazhou et al. applied the improved Inception-Res-Net-V1 network structure to the face recognition problem. This model was trained using the CASIA dataset and the accuracy in the LFW dataset reached 99.22%. Peng Xianlin et al. proposed a multi-task learning method for face expression recognition. This model uses a double-layer tree classifier to replace the Softmax classifier based on the CNN network model, realizing the synchronous classification of face expression labels and face labels, and improving the recognition rate of face expressions. At the same time, the model was trained using the CK+ face expression dataset with an accuracy of 96.62%. Compared with traditional recognition methods, using deep neural networks for license plate recognition can automatically extract the features of license plate images with high accuracy. At present, deep learning technology is the most widely used method in license plate recognition research.In June 2018, Chen Li implemented a license plate recognition algorithm based on the FasterR-CNN model. The algorithm uses FasterR-CNN to locate the license plate image, and then uses the vertical projection method and template matching method to achieve the character segmentation and character recognition of the license plate. The algorithm uses 16,217 license plate images as a data set for training, with an accuracy rate of 95.3%. In September 2018, Cao Zhengfeng et al. constructed a license plate recognition model with a convolutional layer, RPN network, ROI pooling, full connection and RNN network based on the CNN network model, which can directly process the license plate image and eliminate the character segmentation step. The model uses 30,000 constructed vehicle images as a data set for training, with an accuracy rate of 95.4%. In October 2018, Li Xiaoran proposed a tilted license plate correction recognition method, using the CNN network to process the license plate recognition image, and using the Hough transform method to correct the tilted license plate image. The model was trained with 150 blurred tilted license plate images, with an accuracy rate of 98.5%. Deep learning technology can be applied to many aspects of medical imaging, such as CAD (Computer Aided Diagnosis), image segmentation, and image matching and fusion. In 2014, Roth et al. used CNN network to identify clinical lymph node lesions and improved the classification performance of CNN by random rotation sampling. The model trained 388 mediastinal lymph nodes and 595 abdominal lymph nodes, and the prediction accuracy reached 83%. In 2015, Gao combined CNN network with SVM classifier to diagnose the degree of nuclear cataract lesions. The model trained 5378 crowd images with an accuracy of 99.1%. In 2016, Dou proposed a 3DCNN model for automatic detection of brain microbleed images. The model constructed the SWI-CMB dataset in the Philips medical system for training, with an accuracy of 93.16%. In 2017, the Stanford team realized the recognition and classification of skin cancer through the CNN network model. The model integrated 129,450 clinical images as a dataset, and the diagnostic accuracy exceeded the manual diagnosis accuracy. In 2018, Zhang Zezhong and others combined the AlexNet model with the GoogleNet model to classify gastric cancer images. The model was trained with the data set in the competition and achieved an accuracy rate of 99.4% in gastric cancer image recognition and classification. Summary of the invention

[0004] With the widespread popularity of monitoring systems, the vast majority of places are using monitoring systems, including daily life and work units. Therefore, it is necessary to maximize the utilization of monitoring systems. This patent identifies the application programs used on the current computer monitor monitoring system screen, thereby judging the behavior of the monitored person, whether it meets the specified requirements and whether there are dangerous behaviors. At the same time, whether the application program name can be accurately identified on the monitoring screen with a large number and complex situations is the key research object. The convolutional neural network is a deep learning method developed on the basis of multi-layer neural networks and specially designed for image classification and recognition. It has the characteristics of simple structure, few training parameters and strong adaptability. At the same time, its application fields are very extensive, such as gesture recognition, face recognition, iris recognition, vehicle type recognition, etc. Therefore, the method of using deep learning convolutional neural network is used to process such problems.

[0005] This method provides the following technical solutions for the actual application scenario, and focuses on improving and solving the problems of feature extraction, neural network optimization and difficult recognition in complex situations:

[0006] 1) Feature extraction

[0007] Currently, the application scope of monitoring systems is relatively wide, but there are few monitoring applications for computer monitors. The primary research task of this topic is to identify the application programs on the monitor. Therefore, this part needs to preprocess the dataset of software pictures on the screen, normalize the pictures and then extract the features of the pictures, and classify them according to different situations.

[0008] Screen monitoring has the following characteristics:

[0009] a. When the application program is not full screen, it has the feature of software marginalization;

[0010] b. Most software has the feature of program name in the upper left corner;

[0011] c. The feature that the software color composition is different from the background color composition, etc.;

[0012] 2) Application program recognition method based on the fully convolutional neural network method

[0013] Traditional image processing has always relied on features designed by humans, and such features only describe and represent the low-level edge information in the image, and it is difficult to extract deep-level information. Deep learning uses a data-driven approach and adopts a series of non-linear transformations to extract multi-level and multi-angle features from the original data, so that the obtained features have stronger generalization ability and expression ability, opening up a new direction in the research of image processing, thus meeting the needs of efficient image processing. Therefore, this part uses the method of deep learning to improve the recognition model.

[0014] 3) Enhancement of the recognition method based on complex environments

[0015] During the recognition process of a computer monitor monitoring system, various problems will occur. Combining previous research findings, factors such as the recognition base number and the complexity of the recognition interface have a great impact on the recognition results. Therefore, in response to the problems that occurred during the research process, this recognition method is improved and enhanced.

[0016] For the above application scenarios, this method proposes a software recognition method based on screen monitoring, and optimizes the model used by this method according to the actual scenario, so as to greatly improve the recognition rate and accuracy. This method provides the following improved technical processes and technical solutions.

[0017] This method starts from three aspects: the kernel K-means clustering algorithm based on sampling, the inception improvement based on the YOLOv3 object detection algorithm, and the multi-scale improvement based on the YOLOv3 object detection algorithm, and improves YOLOv3 according to the problems existing in YOLOv3 object recognition.

[0018] Kernel K-means clustering algorithm based on sampling: The K-means clustering algorithm is an algorithm that uses distance for clustering. This algorithm believes that the closer the distance between two target points, the higher the similarity between these two targets. Therefore, distance is used as an index to judge the similarity of targets. Multiple target objects with close distances form clusters. The ultimate goal of this algorithm is to obtain multiple independent and compact clusters, that is, multiple classes. The following gives the specific steps of the K-means clustering algorithm: Randomly select K target points as the initial clustering centers; calculate the Euclidean distance d2(x i ,x j )=||x i -x j || 2 , and divide the feature points to be clustered into K clusters; select the centroid points of each cluster as the new clustering centers; repeat steps 2 and 3 until each clustering no longer changes. The K-means algorithm is simple and fast to implement and has high efficiency in processing large data, but there are still several deficiencies. First, the value of the clustering number K is set artificially, and any deviation in the K value will affect the final output clustering cluster results. Second, the target points of the initial clustering centers are randomly selected, and there is a large degree of contingency in this process, which prolongs the execution time of the algorithm; third, the existence of outliers will have a great impact on the clustering results.

[0019] The idea of the kernel K-means clustering algorithm is to map the data into a high-dimensional space and then perform the operations of the classical K-means clustering algorithm. However, in the algorithm, the initial clustering centers are still randomly selected, which has a large degree of contingency. In response to this point, this method introduces the sampling idea and proposes an improved kernel K-means clustering algorithm based on sampling. Using the operation of sampling the data set to improve the kernel K-means clustering algorithm can effectively get rid of the limitation of the data scale. However, whether the size of the extracted sample set and the setting of the initial categories completely cover the original database determine the clustering performance of this algorithm. Therefore, the algorithm proposed in this method first uses the Leaders clustering method to perform initial clustering on the feature points to be clustered to obtain the initial clustering centers; then multiple samples are obtained according to the initial clustering centers to get sample subsets, and kernel K-means clustering is performed on each sub-sample set; finally, the clustering results of each sub-sample set are integrated, so as to enhance the coverage of the initial clustering setting for the original categories while reasonably controlling the scale of the sampled data, making the calculation of the approximate kernel function more reasonable.

[0020] The Leaders algorithm has advantages in dealing with large data sets because it can complete clustering by only scanning the entire data set once and does not require the target number of categories to be set in advance. The Leaders algorithm is related to density and completes the clustering task by selecting the Leader of each target category. First, set the data set as X, the distance threshold as T, allocate the Leader array K, and initialize the number of Leaders k = 0; if the Leader belongs to a certain category and ‖Leader - x i ‖ holds, then the data xi belongs to this category; if not, then this data enters the calculation of the next category until the classification is completed. However, this algorithm is very sensitive to the order of data input and is prone to the phenomenon that the similarity of targets within a class is greater than the similarity of targets between classes during the clustering process. Therefore, when using the Leaders clustering method for classification, the results will show a large deviation. However, due to the fast speed of this algorithm, it can be used as a preprocessing method for other clustering methods. The kernel K-means clustering algorithm based on sampling proposed in this method uses this algorithm for preprocessing clustering to obtain the initial clustering centers and then performs the sampling operation.

[0021] There is a phenomenon that the similarity of targets within a class is greater than the similarity of targets between classes in the result of the Leaders clustering preprocessing. Therefore, the sampling idea is introduced, and random sampling is performed on each initial clustering center obtained by the Leaders clustering preprocessing. While ensuring the correlation between each sampling sample, the misclassified targets are randomly sampled into sub-samples of different classes, and these targets are divided into the correct regions by re-clustering the sample subsets.

[0022] Improvement of Inception Based on YOLOv3 Object Detection Algorithm: In the YOLOv3 model, the DarkNet-53 network is used for the operation of the feature extraction step. In the DarkNet-53 network, the convolutional kernels inside the convolutional layer are all of sizes 1×1 and 3×3. Then, an operation of multi-scale fusion is introduced after the feature extraction network, aiming to make the model more sensitive and accurate in detecting and recognizing objects of different scales. See the following for details Figure 1 , which is the flowchart of the YOLOv3 model. However, the DarkNet-53 feature extraction network has a relatively deep depth and a large number of parameters in the network. Excessive parameters will consume a large amount of time during the training and testing processes. Therefore, this method proposes to introduce the inception idea of the GoogleNet network into the YOLOv3 model. The inception structure used in this method is as Figure 2 shown. It can be seen from the figure that in the inception structure, the 1×1, 3×3, 5×5 and pooling operations are stacked together to stretch the network horizontally, increasing the width of the network and enhancing the network's transmission ability for the upper-layer output. At the same time, convolutional kernels of different sizes make the model more sensitive to objects of different scales. Secondly, in the inception structure, a convolutional layer with a convolutional kernel of 1×1 and a stride of 2 is added to reduce the dimension of the input of the previous layer. This way of reducing the dimension by the convolutional layer can better obtain the global information of the output of the previous layer, while the way of reducing the dimension by the pooling layer will lose some object information. This way of reducing the dimension reduces the network parameters by about 3.9 times.

[0023] Therefore, this method proposes to introduce an inception structure on the basis of the YOLOv3 model. In addition to the receptive fields of sizes 1×1 and 3×3, a receptive field of size 5×5 is added, and a partial residual connection structure (Residual structure) of the DarkNet-53 network is retained, which not only maintains the due depth of the network but also appropriately broadens the width of the network.

[0024] Multi-scale Improvement Based on the YOLOv3 Object Detection Algorithm: The YOLOv3 algorithm draws on the anchor mechanism of the Faster R-CNN model. The number of anchors is pre-set manually, and the initial value set will affect the accuracy of model training and testing. During the training process, the algorithm uses the K-means clustering method to perform clustering analysis on the target bounding boxes. The network will automatically annotate the candidate bounding boxes of the targets according to the target information. The K-means clustering algorithm mainly uses the Euclidean distance to judge the correlation. This means that larger candidate bounding boxes will have a higher false detection rate than smaller ones. Therefore, appropriately increasing the number of anchors will, to a certain extent, solve this problem. Therefore, this method proposes to increase the pre-set number of anchors in the model by three, which can make the classification effect of the model better.

[0025] In different actual scenarios or datasets, targets of the same category also have different sizes. Due to the difference in the distance between the image acquisition device and the target, there are differences in the target scales in the final image set. To make the test model more sensitive to targets of different scales, the YOLOv3 model introduces a multi-scale fusion method for training on the dataset. The YOLOv3 model uses 3 feature maps of different scales for fusion training. However, it is not difficult to find that the shallow feature information in the front part of the DarkNet-53 network is not fully utilized. After passing through multiple convolutional layers, the shallow feature information will be lost to some extent. Therefore, this method proposes that, on the basis of the three-scale fusion of the original YOLOv3 model, add another scale to utilize the shallower information extracted by the network, hoping to improve the detection accuracy of multi-scale targets or small targets.

[0026] The improved algorithm proposed in this method takes into account the feature information extracted by shallower neural networks on the basis of the original model. Upsampling operation is performed on the feature map output by the existing third scale, and then it is fused with the feature map output by the second convolutional layer of the DarkNet-53 network as the fourth-scale branch. Description of the Drawings

[0027] Figure 1 is a schematic diagram of the YOLOv3 model process;

[0028] Figure 2 is a schematic diagram of the Dark_inception network process used in the present invention;

[0029] Figure 3 is a schematic diagram of the Dark_inception network structure in the present invention;

[0030] Figure 4 is a schematic diagram of the improved multi-scale Dark_inception network structure in the present invention;

[0031] Figure 5 It is a schematic diagram for comparing the recognition efficiency of various categories of the three models in the present invention;

[0032] Figure 6 It is a recognition result diagram for two occluded targets in this method; Detailed implementation manners

[0033] The kernel K-means clustering algorithm based on sampling first uses the Leaders algorithm for initial clustering preprocessing to obtain the initial clustering centers; then randomly samples the targets within each initial clustering range, and the misclassified target points are distributed to different sample subsets through random sampling. Now assume the sample subset X = {x1, x2,..., x n}, and its objective function is: where, H k = span{K(x1,*), K(x2,*),..., K(x n ,*)}. Each time when sampling the data near the initial clustering centers, it is necessary to ensure that the union of the sample subsets covers all the categories of the original data set as large as possible, so that the categories included in the union of the sample subsets are as close as possible to the categories of the original data; at the same time, in order to ensure the efficient operation of the kernel K-means clustering algorithm, it is desired that the data volume of each sampled sample subset is small enough. This makes the data volume of each sample subset small, so that parallel computing can be performed to speed up the calculation; however, small sample subsets will also cause a certain deviation between the clustering centers of individual sample subsets and the clustering centers of the original large data. To solve this problem, after performing the kernel K-means clustering algorithm operation on each sample subset, it is necessary to integrate the clustering results of all sample subsets as the final output result.

[0034] First, a non-linear mapping is used to map the known samples in the original space to the high-dimensional kernel space G. In the kernel space, the sample features of different categories are more prominent, making the task become a linear task or an approximately linear task; then, the general K-means clustering algorithm operation is performed in this high-dimensional kernel space to minimize the objective function J: where:

[0035]

[0036] The definition of the kernel distance formula for calculating two feature points in the kernel space is as follows:

[0037]

[0038] After clustering all the sample subsets, all the samples are merged. Then the union of the sample subsets contains K target categories, and the mean value of each category is calculated:

[0039]

[0040] Then calculate the distance between the means of any two classes:

[0041] l = |x i - x j | 2

[0042] If the distance between the means of two target classes is less than a preset threshold, then after merging these two target classes into one class, continue to calculate the class mean distance according to the above formula. After merging and fusing the union of the sample subsets, the final clustering result is obtained.

[0043] Through experiments, it can be verified that the improved algorithm proposed by this method is superior to the classical K-means clustering algorithm. Therefore, in the YOLOv3 model improved by this method, the improved sampling kernel K-means clustering algorithm is used to determine the initial position of the anchor.

[0044] The inception improvement based on the YOLOv3 object detection algorithm makes improvements to the feature extraction part of the YOLOv3 model: using the inception structure to widen the width of the network, increasing the receptive field of the 5×5 size, changing the input during the multi-scale operation of the YOLOv3 model, making the final output result more accurate, and improving the performance of the network. The basic idea is to use different feature maps for location and classification prediction. Since the depth of the YOLO network itself and the semantic features of the feature maps are relatively high, the dilated convolutional feature maps are used for location prediction. At the same time, because more detailed information is required for location prediction, the low-level and high-resolution features are fused to improve the location prediction effect, and the feature maps used for classification and prediction are no longer used for feature fusion. Next Figure 3The figure shows the structural schematic diagram of the Dark_inception network. As can be seen from the red-marked part in the figure, the Dark_inception network improves the entire DarkNet-53 network. It retains the convolutional layer structures of the previous groups, removes the last group of convolutional layer structures, and replaces them with the inception structure. The inception structure in this paper includes receptive fields of three different sizes: 1×1, 3×3, and 5×5, and also includes pooling. The number of channels is to ensure that the finally output feature map has a size of 13×13×1024. Then there is still the multi-scale fusion operation. The first scale is the newly generated feature map of 13×13×1024; the second scale is the feature map of 26×26×512 obtained after upsampling the new feature map of 13×13×1024, which is connected to the feature map of 26×26×256 of the output layer of the second-to-last group of the network; the third scale is the feature map of 52×52×256 obtained after upsampling again, and then it is connected to the feature map of 52×52×218 of the output layer of the second-to-last group of the network. The improved network is deeper in width than before the improvement in structure, reducing the computational amount, enabling the input layer of the model's multi-scale operation to obtain more global target information, and fusing with the feature map output by the shallow layer. Therefore, the final detection effect is also improved.

[0045] Figure 4 The figure shows the structural schematic diagram of the improved multi-scale Dark_inception network. The multi-scale Dark_inceptio network proposed in this method is improved on the basis of the above-mentioned Dark_inception network. It adds the inception structure on the basis of YOLOv3 and adds a fourth scale to improve the model. The improvement points of the model are marked with red rectangular frames in the figure. It can be seen that the model retains the original DarkNet-53 feature extraction network of YOLOv3, adds a branch at the output of the third scale, performs upsampling on the feature map of size 52×52×128 output by the third scale, and then fuses it with the feature map of size 104×104×128 output by a shallower convolutional network to perform the fourth-scale fusion. There are 53 convolutional layers in the original DarkNet-53 feature extraction network. After multiple convolutional layers, some of the feature information extracted by the shallow network will be lost. After the improvement of the fourth scale, it fully considers the feature information of the shallow network for training the model. In this way, the trained model will be more sensitive to multi-scale targets and the accuracy will be greatly improved.

[0046] To visually reflect the improvement effect of this method, different screen monitoring video frames are selected for recognition verification and comparison in this method. The targets in the dataset are recognized to obtain the comparison results as shown in Figure 5 . The results show that the average accuracy of the improved model is higher than that of the classic YOLOv3 network. In addition, this method verifies the situation of overlapping and occlusion in screen monitoring and obtains good recognition results.

[0047] In the description of the present invention, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0048] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be embraced within the present invention; any reference signs in the claims should not be regarded as limiting the claims involved.

[0049] In addition, it should be understood that although this specification is described according to the embodiments, not every embodiment only contains an independent technical solution. This narrative way of the specification is only for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A software recognition method based on screen monitoring, characterized in that: Step S1: Preprocess the dataset using the Leaders clustering algorithm, then perform a sampling operation to extract several sample subsets. Conduct kernel K-means clustering on the sample subsets, and finally integrate the clustering results by calculating the mean distance between each sample subset to determine the initial position of the anchor, which is then fed into the YOLOv3 model for object detection; Step S2: Introduce the inception structure used to build a sparse network, making the network layer with complete parameter connections become a sparse network layer, reducing the number of training parameters while widening the width of the network; Step S3: Improve the multi-scale fusion operation of the YOLOv3 model, consider the target information in the shallower layer of the feature extraction network, and enhance the network's sensitivity to small targets; The said Step S1 includes: First, use the Leaders clustering method to perform initial clustering on the feature points to be clustered to obtain the initial clustering centers; then, obtain sample subsets through multiple samplings based on the initial clustering centers, and perform kernel K-means clustering on each sub-sample set; finally, integrate the clustering results of each sub-sample set, thereby enhancing the coverage of the initial clustering settings for the original categories while reasonably controlling the scale of the sampled data, making the calculation of the approximate kernel function more reasonable; The said Step S2 includes: Use the inception structure to widen the width of the network, increase the receptive field of 5×5 size, change the input during the multi-scale operation of the YOLOv3 model, make the final output result more accurate, and improve the network performance; The said Step S3 includes: On the basis of the three-scale fusion of the original YOLOv3 model, add another scale to utilize the shallower information extracted by the network, which can enhance the detection accuracy of multi-scale targets or small targets.

Images