Software interface element detection method and system based on multi-granularity learning, computer equipment and storage medium

A software interface and detection method technology, applied in software testing/debugging, computing, error detection/correction, etc., can solve problems such as low efficiency, human factors, error-prone, etc., to reduce interference, ensure detection effectiveness, The effect of improving efficiency and accuracy

CN111104339AActive Publication Date: 2020-05-05SHANGHAI YISAIQI SOFTWARE CO LTD
7 Cites 4 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Publication Date
2020-05-05

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention discloses a software interface element detection and positioning method and system based on multi-granularity learning, computer equipment and a storage medium. The method comprises thesteps of: obtaining an operation interface image; inputting the image into a deep convolutional neural network to extract features to obtain the intermediate result of the same size; performing meshing division on the intermediate result; inputting the intermediate result into a multi-granularity cascade background discriminator, and carrying out background grid detection on the intermediate imageby a detector by taking a grid as a unit; after background grids are removed, partially inputting the remaining grids into a multi-granularity software interface element detection model for softwareinterface element detection, and outputting the type and the area size of a control to be detected. According to the multi-granularity cascaded background grid detection device provided by the invention, the background grid can be quickly and effectively found out, and the efficiency and accuracy of software interface element detection and positioning are improved.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention relates to a multi-granularity-based software interface element detection method, system, computer equipment and storage medium, and belongs to the field of automation software auxiliary technology. Background technique

[0002] Automatic detection of software interface elements is the key technology of software intelligence. Software intelligence and some software auxiliary fields need to be able to automatically detect software interface elements. However, with the update iteration of software interface and the continuous enrichment of software interface element types, the complexity of automatic control detection is also increasing.

[0003] In the process of software development and testing, it is also necessary to detect the software interface, that is, to detect whether the software interface is consistent with the software interface in the design drawing. At present, it is mainly necessary to rely on testers to check whether the s...

Examples

Embodiment example

[0079] Step (1), input the image of the software interface; input an image, the image is an RGB three-channel image, and the size is W*H.

[0080] Step (2), use the convolutional neural network to obtain the features of the image, and obtain an intermediate result of the same size; the intermediate result is an image of D*W*H.

[0081] Step (3), perform multi-granularity meshing operation on the features obtained in step (2); suppose the granularity size is (a 1 *b 1 ),(a 2 *b 2 ), (a 3 *b 3 )……(a n *b n ), then n groups of image grids of different sizes are obtained respectively. In this embodiment, assuming that the granularity is (4x4), (8x8), (16x16), three sets of grids can be obtained: the first set of 16 grids (4x4 equal parts of the original image), the second set of 64 grids, and the third group of 256 grids.

[0082] Step (4), use a multi-grain cascade background discriminator to calibrate the non-control background grid area of ​​the software interface; ass...