A UI control recognition method, device, equipment and computer readable storage medium

Abstract

The application discloses a UI control recognition method and device, equipment and a computer readable storage medium. A target image deep learning network is trained through a preset training set and a random training set, so that position information, number information, category information and a prediction score of a small cutout or a content image in a development page and a design page can be obtained, a first development recognition result and a first design recognition result are determined, a first comparison result is obtained, and the UI control recognition method and device have the capability of automatically detecting and recognizing the control, and solve the technical problem that it is difficult to compare the development page and the design page in the case of a complex attribute element page.

Description

Technical Field

[0001] This application relates to the field of computer front-end technology, and in particular to a UI control recognition method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] UI design (or interface design) refers to the overall design of software's human-computer interaction, operation logic, and interface aesthetics. When comparing the UI of developed and designed pages to see if they are the same, it is usually necessary to identify the information of the controls before comparison. Among them, comparing UI combination controls is a method that can reduce comparison errors.

[0003] The traditional method is to obtain web page elements from the development page through pure HTML. However, web page elements are usually complex and varied in structure and depend on reasonable naming conventions such as class names and fixed styles for web page elements. At the same time, for mobile apps developed based on native technology, it is not possible to obtain page element information by obtaining web page elements. This leads to technical problems in comparing the development page and the design page when there are complex attribute elements. Summary of the Invention

[0004] This application provides a UI control identification method, apparatus, device, and computer-readable storage medium, which solves the technical problem of difficulty in comparing development pages and design pages in the case of pages with complex attribute elements.

[0005] In view of the above, the first aspect of this application provides a UI control recognition method, the method comprising:

[0006] S1. Construct first training data. The first training data is an XML file containing image information of several first training images. Each first training image contains several small cut-out images or content images. All small cut-out images or content images are labeled. The file name of the XML file corresponds to the image name of the corresponding first training image.

[0007] S2. Divide the first training data into a training set and a test set according to a preset ratio, and randomly extract XML files from the training set and the test set to generate a random training set and a random test set;

[0008] S3. Obtain the image file name of the corresponding first training image according to the training set, the test set, the random training set and the xml file in the random test set, and save the image file name of the first training image after removing the suffix as an index to a txt file, so as to obtain the training txt file, the test txt file, the random training txt file and the random test txt file respectively.

[0009] S4. Load the training set and the XML file in the random training set as input data for K-means clustering, train the target image deep learning network, and test the target image deep learning network with the test set and the random test set to obtain the target image deep learning network with the location information, number information, category information and prediction score of small slices or content images as output data.

[0010] S5. Based on the target image deep learning network, image control recognition is performed on the development page and the design page respectively to obtain the first development recognition result and the first design recognition result respectively;

[0011] S6. Compare the first development identification result with the first design identification result to obtain the first comparison result.

[0012] Optionally, it also includes:

[0013] S7. Construct second training data, which includes several second training images carrying title elements or subtitle elements;

[0014] S8. After performing feature preprocessing on the second training image, a probability map and a threshold map are generated.

[0015] S9. Based on the preset loss function, the DBNet detection network is trained to obtain a target text detection model with the position information of the text as the output data.

[0016] S10. Based on the target text detection model, perform text control recognition on the development page and the design page respectively to obtain the second development recognition result and the second design recognition result respectively.

[0017] S11. Compare the second development identification result with the second design identification result to obtain the second comparison result.

[0018] Optionally, it also includes:

[0019] S12. Combine the first comparison result with the second comparison result to determine the position information of the graphic and text combination control.

[0020] Optionally, step S12 specifically includes:

[0021] Based on the position information of the small cut image or content image corresponding to the image control information and text control information on the same target page, as well as the position information of the text, the positional relationship between the small cut image or content image and the text is determined. The small cut image or content image and the text are merged into a graphic control, and the position information of the graphic control is determined based on the positional relationship.

[0022] Optionally, step S4 specifically includes:

[0023] Load the training set and the XML file in the random training set, and obtain the minimum x-coordinate, minimum y-coordinate, maximum x-coordinate, and maximum y-coordinate of the region position of the small slice or content image;

[0024] Based on the minimum x-coordinate, minimum y-coordinate, maximum x-coordinate, and maximum y-coordinate of the region location of the small slice or content map, the width and height of the ground truth bounding box are used as input data for K-means clustering to train the deep learning network for the target image.

[0025] The target image deep learning network is tested using the test set and the random test set to obtain a target image deep learning network that outputs location information, number information, category information, and prediction score of small slices or content maps, wherein the prediction score is the product of confidence and category probability.

[0026] A second aspect of this application provides a UI control recognition device, the device comprising:

[0027] The first construction unit is used to construct the first training data, which is an XML file containing image information of several first training images. The first training images contain several small cut images or content images, and the small cut images or content images are all labeled. The file name of the XML file corresponds to the image name of the corresponding first training image.

[0028] The first preprocessing unit is used to divide the first training data into a training set and a test set according to a preset ratio, and randomly extract xml files from the training set and the test set to generate a random training set and a random test set.

[0029] The second preprocessing unit is used to obtain the image file name of the corresponding first training image according to the training set, the test set, the random training set and the xml file in the random test set, and save the image file name of the first training image after removing the suffix as an index to a txt file, thereby obtaining a training txt file, a test txt file, a random training txt file and a random test txt file respectively.

[0030] The first training unit is used to load the training set and the XML file in the random training set as input data for K-means clustering, train the target image deep learning network, and test the target image deep learning network with the test set and the random test set to obtain the target image deep learning network with the location information, number information, category information and prediction score of small slices or content maps as output data.

[0031] The image recognition unit is used to perform image control recognition on the development page and the design page based on the target image deep learning network, and obtain the first development recognition result and the first design recognition result respectively.

[0032] The first comparison unit is used to compare the first development identification result with the first design identification result to obtain the first comparison result.

[0033] Optionally, it also includes:

[0034] The second construction unit is used to construct the second training data, which includes several second training images carrying title elements or subtitle elements.

[0035] The third preprocessing unit is used to perform feature preprocessing on the second training image to generate a probability map and a threshold map.

[0036] The second training unit is used to train the DBNet detection network based on a preset loss function to obtain a target text detection model with the position information of the text as the output data.

[0037] The text recognition unit is used to perform text control recognition on the development page and the design page respectively based on the target text detection model, and obtain the second development recognition result and the second design recognition result respectively.

[0038] The second comparison unit is used to compare the second development identification result with the second design identification result to obtain the second comparison result.

[0039] Optionally, it also includes:

[0040] The combined recognition unit is used to combine the first comparison result with the second comparison result to determine the position information of the graphic and text combination control.

[0041] A third aspect of this application provides a UI control recognition device, the device comprising a processor and a memory:

[0042] The memory is used to store program code and transmit the program code to the processor;

[0043] The processor is configured to execute the steps of the UI control recognition method as described in the first aspect above, according to the instructions in the program code.

[0044] A fourth aspect of this application provides a computer-readable storage medium for storing program code for performing the method described in the first aspect above.

[0045] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0046] This application provides a UI control recognition method, apparatus, device, and computer-readable storage medium. By training a deep learning network on a target image using a preset training set and a random training set, it is possible to obtain the position information, number information, category information, and predicted scores of small slices or content images in the development page and the design page, thereby clarifying the first development recognition result and the first design recognition result, and obtaining the first comparison result. It has the ability to automatically detect and recognize controls, and solves the technical problem of difficulty in comparing development pages and design pages in the case of pages with complex attribute elements. Attached Figure Description

[0047] Figure 1 This is the first method flowchart of the UI control recognition method in the embodiments of this application;

[0048] Figure 2 This is a flowchart of the second method of the UI control recognition method in the embodiments of this application;

[0049] Figure 3 This is a first structural schematic diagram of the UI control recognition device in the embodiments of this application;

[0050] Figure 4 This is a second structural schematic diagram of the UI control recognition device in the embodiments of this application;

[0051] Figure 5 This is a schematic diagram of the structure of the UI control recognition device in the embodiments of this application. Detailed Implementation

[0052] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0053] This application presents a UI control identification method, apparatus, device, and computer-readable storage medium, which solves the technical problem of difficulty in comparing development pages and design pages in the case of pages with complex attribute elements.

[0054] For easier understanding, please refer to Figure 1 , Figure 1 This is the first method flowchart of the UI control recognition method in the embodiments of this application, as follows: Figure 1 As shown, specifically:

[0055] S1. Construct the first training data. The first training data is an XML file containing image information of several first training images. Each first training image contains several small cut-out images or content images. All small cut-out images or content images are labeled. The file name of the XML file corresponds to the image name of the corresponding first training image.

[0056] It should be noted that the detection of image elements includes two types: one is the small control slices usually cut out by UI designers, and the other is the content images that change, i.e., the images loaded from the server. First, for the small control slices, we collected 320 small slices from mobile and web platforms, and 1000 content images as one category.

[0057] Construct the first training data, in which each image contains multiple small cut-out images and multiple content images. Save these images in the same folder and use the open-source labelImg tool to label them. Labeling involves outlining the small cut-out images or content images and their object names, such as homeBtn. Then save the data as an XML file.

[0058] The XML file contains the name of the custom image object, such as photo1, the image file name, the image file path, the image size corresponding to width, height, and depth, the minimum x-coordinate, minimum y-coordinate, maximum x-coordinate, and maximum y-coordinate of the small slice or content image area, and the object name of the small slice or content image, such as homeBtn.

[0059] These XML files are placed in another folder. The files in the image folder and the files in the XML folder are linked one-to-one by filename. Similarly, the method of tagging other small cut images and content images, as well as the method of saving the files, are the same as those of tagging small cut images or content images. After tagging the same image, the content is saved in the same XML file. The object name tags of the corresponding small cut images and content images can also be represented by 0 and 1. Here, English names are used, such as homeBtn, etc.

[0060] S2. Divide the first training data into a training set and a test set according to a preset ratio, and randomly extract XML files from the training set and the test set to generate a random training set and a random test set.

[0061] S3. Obtain the image file name of the corresponding first training image from the XML file in the training set, test set, random training set and random test set, and save the image file name of the first training image to a txt file after removing the suffix, so as to obtain the training txt file, test txt file, random training txt file and random test txt file respectively.

[0062] It should be noted that the dataset is divided into training and test sets according to a preset ratio. After confirming the number of partitions, the image filenames are obtained by randomly selecting XML files, removing the suffix (e.g., .jpg), and then saved as indexes to a TXT file in single-line format. This results in the file containing test.txt, train.txt, trainval.txt, and val.txt. Therefore, by using the indexes in the TXT file, the corresponding labels from the XML files and the original images can be retrieved simultaneously.

[0063] S4. Load the training set and the XML file in the random training set as input data for K-means clustering, train the target image deep learning network, and test the target image deep learning network with the test set and the random test set to obtain the target image deep learning network with the location information, number information, category information and prediction score of small cut maps or content maps as output data.

[0064] It should be noted that during the training process, the XML file in the XML folder is first loaded to obtain the minimum x-coordinate, minimum y-coordinate, maximum x-coordinate, and maximum y-coordinate of the location of each small slice and content image region for labeling. Then, the width and height of the ground truth bounding box are used as features as input data for K-means clustering. Considering that the size of each bounding box is different in different scenes, it is very necessary to standardize the width and height of the bounding box to match the width and height of the image.

[0065] The deep learning network structure used for training the target images is as follows:

[0066] Based on the YOLO framework, CSP is added to each large residual block of Darknet53, corresponding to layers 0 to 104, thus forming the backbone network. Spatial pyramid pooling is added to increase the receptive field of the network. Max pooling of layer 107 is performed at 5*5, 9*9, and 13*13, resulting in layers 108, 110, and 112, respectively. After pooling, they are concatenated to form the feature map layer 114 and reduced to 512 channels by 1*1. Based on FPN, upsampling is followed by downsampling to achieve feature fusion.

[0067] The training was conducted using four M60 GPUs. The image size input to the deep network was adjusted to 416*416 according to the available GPU memory. The image was input into the deep network structure shown above. The training model parameters included Mosaic data augmentation, label smoothing, CIOU learning rate cosine annealing decay, and Mish activation function. Additionally, during training, the backbone feature extraction network features were frozen to speed up the training process and prevent weights from being corrupted in the early stages. The training lasted for 200 epochs. The initial learning rate was set to le-3 and the batch size to 4 for the first 100 epochs. For the last 100 epochs, the initial learning rate was set to le-4 and the batch size to 2 after some experimentation to speed up the training and reduce GPU memory usage.

[0068] The result is constructed as follows: In this implementation, confidence is the probability that a bounding box contains an object and the accuracy of its location (i.e., whether it exactly encloses a small cut or content map region). Its formula is Pr(hid) * IOU, where IOU is the IOU between the predicted and the true value. During training, the Pr(hid) * IOU of the label is 1. During prediction, confidence is the predicted value, representing the above meaning. When predicting whether something is a small cut (the object name of a small cut region, such as homeBtn), there is a conditional probability value related to the class, i.e., the class probability under the confidence. Therefore, the final scores are confidence * class probability. After initial screening by class probabilities greater than a preset parameter of 0.5, all prediction results under all classes are processed. The remaining prediction results are then sorted by confidence * class probability. The highest score is taken from the sorted results, and non-maximum suppression is applied to remove overlap values ​​greater than a preset parameter of 0.4 to obtain the final optimal prediction results for each class.

[0069] The output is (x1, y1, x2, y2, obj_conf, class_conf, class_pred), representing the minimum x-coordinate, minimum y-coordinate, maximum x-coordinate, maximum y-coordinate, object confidence, class probability, and class, respectively. The number of small cut plots indicates the number of results for finding small cut plot regions belonging to a specific class, such as homeBtn. Their positions can be directly obtained from the results above. The final scores are obj_conf * class_conf in the results.

[0070] Based on the above, we obtained the regression positions and number of small cut plots or content plots, as well as the predicted scores.

[0071] S5. Based on the target image deep learning network, image control recognition is performed on the development page and the design page respectively to obtain the first development recognition result and the first design recognition result respectively.

[0072] S6. Compare the first development identification result with the first design identification result to obtain the first comparison result.

[0073] Further, please refer to Figure 2 , Figure 2 This is a flowchart of the second method of the UI control recognition method in the embodiments of this application, as follows: Figure 2 As shown, it also includes:

[0074] S7. Construct the second training data, which includes several second training images carrying title elements or subtitle elements;

[0075] It should be noted that the second training data consists of 190,000 images from publicly available open-source data, various financial business scenario data, and synthetic data. When the model is inferring, the maximum size of the input image is set to 960 with proportional scaling. The detection filtering threshold is set to 0.5, and text boxes below the threshold are filtered out. After text detection, the Aboxes of each position in the text line are saved. For text block recognition, a CRNN network is used.

[0076] S8. After performing feature preprocessing on the second training image, a probability map and a threshold map are generated.

[0077] It should be noted that the features at 1 / 32, 1 / 16, 1 / 8, and 1 / 4 of the image are fused into the original. Figure 1 A feature map of size 4 is used as a fuse, and then the fuse is deconvolved to the original image size to generate a probability map (i.e., a segmentation map) and a threshold map.

[0078] S9. Based on the preset loss function, the DBNet detection network is trained to obtain a target text detection model with the position information of the text as the output data.

[0079] It should be noted that the model classifies and detects title and subtitle elements; it adds a classification branch to the DBNet detection network's DB segmentation head.

[0080] Its text detection model, DBNet (Segmentation-Based Differentiable Binarized Text Detection Network), has a basic model structure of FPN. Its loss function is L = Ls + α × Lb + β × Lt, where Ls is the loss of the probability map, Lb is the loss of the binary map, and Lt is the loss of the threshold map. Ls and Lb are calculated using BCE loss (binary cross-entropy). To balance positive and negative samples, an online hard example mining strategy is adopted, with a positive-to-negative sample ratio of 1:3. Lt is the L1 loss, α and β are 1.0 and 10 respectively, y* represents the label of the threshold map, and Rd represents the set of pixels within the dilated text polygon. Finally, the positional information of the text element (x, y, w, h) is obtained, where x represents the minimum horizontal coordinate, y represents the minimum vertical coordinate, w is the width, and h is the height.

[0081] S10. Based on the target text detection model, perform text control recognition on the development page and the design page respectively to obtain the second development recognition result and the second design recognition result respectively.

[0082] S11. Compare the second development identification result with the second design identification result to obtain the second comparison result.

[0083] Furthermore, it also includes:

[0084] S12. Combine the first comparison result with the second comparison result to determine the position information of the graphic and text combination control.

[0085] Furthermore, step S12 specifically includes:

[0086] Based on the position information of the small cutout or content image corresponding to the image control information and text control information on the same target page, as well as the position information of the text, the positional relationship between the small cutout or content image and the text is determined. The small cutout or content image and the text are merged into a graphic control, and the position information of the graphic control is determined based on the positional relationship.

[0087] It should be noted that the alignment of graphic elements in design and development is divided into left image and right text, top image and bottom text, and text within an image.

[0088] If the minimum ordinate of the image and the minimum ordinate of the title element are kept consistent, and the minimum difference in ordinate is set to within 5 pixels in height, and the title element is to the right of the image element, then it is judged as a left-image-right-text scenario.

[0089] If there is a subtitle element, determine that the subtitle is below the title and does not exceed the maximum value of the vertical coordinate of the image element, and merge the left image and right text into a combo control. Obtain the position information of the combo control based on the minimum and maximum horizontal coordinates and the minimum and maximum total coordinates of the image and text.

[0090] If the minimum horizontal coordinate of the image and the minimum horizontal coordinate of the title element are kept consistent, and the minimum difference in horizontal coordinates is set to within 5 pixels of height, and the title element is below the image element, then it is judged as an image above text.

[0091] If there is a subtitle, determine that the subtitle is below the title, i.e., the minimum ordinate of the subtitle is greater than the minimum ordinate of the title. Combine the image and text into a single control and obtain the position information of the control based on the minimum and maximum horizontal and vertical coordinates of the image and text.

[0092] If the minimum x-coordinate of the image is less than the minimum x-coordinate of the subtitle, the minimum x-coordinate of the image is less than the minimum x-coordinate of the title, the maximum x-coordinate of the image is greater than the maximum x-coordinate of the subtitle, the maximum x-coordinate of the image is greater than the maximum x-coordinate of the title, the minimum y-coordinate of the image is less than the minimum y-coordinate of the subtitle, the minimum y-coordinate of the image is less than the minimum y-coordinate of the title, the maximum y-coordinate of the image is greater than the maximum y-coordinate of the subtitle, and the maximum y-coordinate of the image is greater than the maximum y-coordinate of the title, then the image is determined to contain text. This image containing text is treated as a combined control, and the position information of the image is the position information of the combined control.

[0093] Furthermore, step S4 specifically includes:

[0094] Load the training set and the XML file in the random training set, and obtain the minimum x-coordinate, minimum y-coordinate, maximum x-coordinate, and maximum y-coordinate of the region position of the small slice or content image;

[0095] Based on the minimum x-coordinate, minimum y-coordinate, maximum x-coordinate, and maximum y-coordinate of the region location in the small slice or content map, the width and height of the ground truth bounding box are used as input data for K-means clustering to train a deep learning network for the target image.

[0096] The target image deep learning network was tested using a test set and a random test set to obtain the target image deep learning network with the location information, number information, category information, and prediction score of small slices or content maps as output data. The prediction score is the product of confidence and category probability.

[0097] By detecting and recognizing composite controls, it becomes easier to compare development and design pages. It can also monitor and accurately test the operation and click of composite controls in the development page. In addition, the text recognized for titles and subtitles not only increases the comparison of fixed text between the development page and the design page, enhancing its page comparison capabilities, but also assists in the manual verification of comparison results and the results of automated click tests.

[0098] Please see Figure 3 , Figure 3This is a first structural schematic diagram of the UI control recognition device in the embodiments of this application, as shown below. Figure 3 As shown, specifically:

[0099] The first construction unit 301 is used to construct the first training data. The first training data is an XML file containing image information of several first training images. The first training images contain several small cut images or content images, and the small cut images or content images are all labeled. The file name of the XML file corresponds to the image name of the corresponding first training image.

[0100] The first preprocessing unit 302 is used to divide the first training data into a training set and a test set according to a preset ratio, and randomly extract xml files from the training set and the test set to generate a random training set and a random test set.

[0101] The second preprocessing unit 303 is used to obtain the image file name of the corresponding first training image according to the XML file in the training set, test set, random training set and random test set, and save the image file name of the first training image after removing the suffix as an index to a txt file, so as to obtain the training txt file, test txt file, random training txt file and random test txt file respectively.

[0102] The first training unit 304 is used to load the training set and the XML file in the random training set as input data for K-means clustering, train the target image deep learning network, and test the target image deep learning network with the test set and the random test set, so as to obtain the target image deep learning network with the location information, number information, category information and prediction score of small cut maps or content maps as output data.

[0103] The image recognition unit 305 is used to perform image control recognition on the development page and the design page based on the target image deep learning network, and obtain the first development recognition result and the first design recognition result respectively.

[0104] The first comparison unit 306 is used to compare the first development identification result with the first design identification result to obtain the first comparison result.

[0105] Further, please refer to Figure 4 , Figure 4 This is a second structural schematic diagram of the UI control recognition device in the embodiments of this application, as shown below. Figure 4 As shown, it also includes:

[0106] The second construction unit 401 is used to construct the second training data, which includes several second training images carrying title elements or subtitle elements.

[0107] The third preprocessing unit 402 is used to perform feature preprocessing on the second training image to generate a probability map and a threshold map.

[0108] The second training unit 403 is used to train the DBNet detection network based on a preset loss function to obtain a target text detection model with the position information of the text as the output data.

[0109] The text recognition unit 404 is used to perform text control recognition on the development page and the design page respectively based on the target text detection model, and obtain the second development recognition result and the second design recognition result respectively.

[0110] The second comparison unit 405 is used to compare the second development identification result with the second design identification result to obtain the second comparison result.

[0111] Furthermore, it also includes:

[0112] The combined recognition unit is used to combine the first comparison result with the second comparison result to determine the position information of the graphic and text combination control.

[0113] This application also provides another UI control recognition device, such as... Figure 5 As shown, for ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. The terminal can be any terminal device including mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) terminals, in-vehicle computers, etc. Taking a mobile phone as an example:

[0114] Figure 5 This is a block diagram illustrating a portion of the structure of a mobile phone related to the terminal provided in the embodiments of this application. (Reference) Figure 5 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090, etc. Those skilled in the art will understand that... Figure 5 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0115] The following is combined Figure 5A detailed introduction to each component of a mobile phone:

[0116] The RF circuit 1010 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with the processor 1080; additionally, it transmits uplink data to the base station. Typically, the RF circuit 1010 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, the RF circuit 1010 can also communicate wirelessly with networks and other devices. The aforementioned wireless communications may use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Messaging Service (SMS).

[0117] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 1020. The memory 1020 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 1020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0118] The input unit 1030 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 1031), and drive the corresponding connection devices according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 1080, and can also receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1031, the input unit 1030 may also include other input devices 1032. Specifically, other input devices 1032 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0119] The display unit 1040 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 1040 may include a display panel 1041, which may optionally be configured as a Liquid Crystal Display (LCD), Organic Light-Emitting Diode (OLED), or similar display panel 1041. Further, a touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it transmits the information to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 5 In this embodiment, the touch panel 1031 and the display panel 1041 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0120] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 1041 according to the ambient light level, and the proximity sensor can turn off the display panel 1041 and / or the backlight when the phone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, taps), etc. Other sensors that may be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0121] The audio circuit 1060, speaker 1061, and microphone 1062 provide an audio interface between the user and the mobile phone. The audio circuit 1060 converts the received audio data into electrical signals and transmits them to the speaker 1061, where the speaker 1061 converts them into sound signals for output. On the other hand, the microphone 1062 converts the collected sound signals into electrical signals, which are then received by the audio circuit 1060, converted into audio data, and then processed by the processor 1080 before being transmitted via the RF circuit 1010 to, for example, another mobile phone, or the audio data can be output to the memory 1020 for further processing.

[0122] WiFi is a short-range wireless transmission technology. Through the WiFi module 1070, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 5 The WiFi module 1070 is shown, but it is understood that it is not an essential component of a mobile phone and can be omitted as needed without changing the essence of the invention.

[0123] The processor 1080 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It executes software programs and / or modules stored in the memory 1020 and calls data stored in the memory 1020 to perform various functions and process data, thereby providing overall monitoring of the phone. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 1080.

[0124] The mobile phone also includes a power supply 1090 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0125] Although not shown, mobile phones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0126] In this embodiment of the application, the processor 1080 included in the terminal also has the following functions:

[0127] S1. Construct the first training data. The first training data is an XML file containing image information of several first training images. Each first training image contains several small cut-out images or content images. All small cut-out images or content images are labeled. The file name of the XML file corresponds to the image name of the corresponding first training image.

[0128] S2. Divide the first training data into a training set and a test set according to a preset ratio, and randomly extract XML files from the training set and the test set to generate a random training set and a random test set.

[0129] S3. Obtain the image file name of the corresponding first training image from the XML file in the training set, test set, random training set and random test set, and save the image file name of the first training image to a txt file after removing the suffix, so as to obtain the training txt file, test txt file, random training txt file and random test txt file respectively.

[0130] S4. Load the training set and the XML file in the random training set as input data for K-means clustering, train the target image deep learning network, and test the target image deep learning network with the test set and the random test set to obtain the target image deep learning network with the location information, number information, category information and prediction score of small cut maps or content maps as output data.

[0131] S5. Based on the target image deep learning network, image control recognition is performed on the development page and the design page respectively to obtain the first development recognition result and the first design recognition result respectively.

[0132] S6. Compare the first development identification result with the first design identification result to obtain the first comparison result.

[0133] This application provides a UI control recognition method, apparatus, device, and computer-readable storage medium. By training a deep learning network on a target image using a preset training set and a random training set, the method can obtain the location information, number information, category information, and predicted scores of small slices or content images in the development page and the design page. This allows for the determination of the first development recognition result and the first design recognition result, resulting in a first comparison result. The method has the ability to automatically detect and recognize controls, solving the technical problem of difficulty in comparing development pages and design pages in the case of pages with complex attribute elements.

[0134] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0135] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0136] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0138] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0139] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0140] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0141] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for recognizing UI controls, characterized in that, include: S1. Construct first training data. The first training data is an XML file containing image information of several first training images. Each first training image contains several small cut-out images or content images. All small cut-out images or content images are labeled. The file name of the XML file corresponds to the image name of the corresponding first training image. S2. Divide the first training data into a training set and a test set according to a preset ratio, and randomly extract XML files from the training set and the test set to generate a random training set and a random test set; S3. Obtain the image file name of the corresponding first training image according to the training set, the test set, the random training set and the xml file in the random test set, and save the image file name of the first training image after removing the suffix as an index to a txt file, so as to obtain the training txt file, the test txt file, the random training txt file and the random test txt file respectively. S4. Load the training set and the XML file in the random training set as input data for K-means clustering, train the target image deep learning network, and test the target image deep learning network with the test set and the random test set to obtain the target image deep learning network with the location information, number information, category information and prediction score of small slices or content images as output data. S5. Based on the target image deep learning network, image control recognition is performed on the development page and the design page respectively to obtain the first development recognition result and the first design recognition result respectively; S6. Compare the first development identification result with the first design identification result to obtain the first comparison result; S7. Construct second training data, which includes several second training images carrying title elements or subtitle elements; S8. After performing feature preprocessing on the second training image, a probability map and a threshold map are generated. S9. Based on the preset loss function, the DBNet detection network is trained to obtain a target text detection model with the position information of the text as the output data. The loss function of the target text detection model is L = Ls + α×Lb + β×Lt, where Ls is the loss of the probability map, Lb is the loss of the binary map, and Lt is the loss of the threshold map. Ls and Lb are solved using BCE loss binary cross-entropy. S10. Based on the target text detection model, perform text control recognition on the development page and the design page respectively to obtain the second development recognition result and the second design recognition result respectively. S11. Compare the second development identification result with the second design identification result to obtain the second comparison result.

2. The UI control recognition method according to claim 1, characterized in that, Also includes: S12. Combine the first comparison result with the second comparison result to determine the position information of the graphic and text combination control.

3. The UI control recognition method according to claim 2, characterized in that, Step S12 specifically includes: Based on the position information of the small cut image or content image corresponding to the image control information and text control information on the same target page, as well as the position information of the text, the positional relationship between the small cut image or content image and the text is determined. The small cut image or content image and the text are merged into a graphic control, and the position information of the graphic control is determined based on the positional relationship.

4. The UI control recognition method according to claim 1, characterized in that, Step S4 specifically includes: Load the training set and the XML file in the random training set, and obtain the minimum x-coordinate, minimum y-coordinate, maximum x-coordinate, and maximum y-coordinate of the region position of the small slice or content image; Based on the minimum x-coordinate, minimum y-coordinate, maximum x-coordinate, and maximum y-coordinate of the region location of the small slice or content map, the width and height of the ground truth bounding box are used as input data for K-means clustering to train the deep learning network for the target image. The target image deep learning network is tested using the test set and the random test set to obtain a target image deep learning network that outputs location information, number information, category information, and prediction score of small slices or content maps, wherein the prediction score is the product of confidence and category probability.

5. A UI control recognition device, characterized in that, include: The first construction unit is used to construct the first training data, which is an XML file containing image information of several first training images. The first training images contain several small cut images or content images, and the small cut images or content images are all labeled. The file name of the XML file corresponds to the image name of the corresponding first training image. The first preprocessing unit is used to divide the first training data into a training set and a test set according to a preset ratio, and randomly extract xml files from the training set and the test set to generate a random training set and a random test set. The second preprocessing unit is used to obtain the image file name of the corresponding first training image according to the training set, the test set, the random training set and the xml file in the random test set, and save the image file name of the first training image after removing the suffix as an index to a txt file, thereby obtaining a training txt file, a test txt file, a random training txt file and a random test txt file respectively. The first training unit is used to load the training set and the XML file in the random training set as input data for K-means clustering, train the target image deep learning network, and test the target image deep learning network with the test set and the random test set to obtain the target image deep learning network with the location information, number information, category information and prediction score of small slices or content images as output data. The image recognition unit is used to perform image control recognition on the development page and the design page based on the target image deep learning network, and obtain the first development recognition result and the first design recognition result respectively. The first comparison unit is used to compare the first development identification result with the first design identification result to obtain the first comparison result.

6. The UI control recognition device according to claim 5, characterized in that, Also includes: The second construction unit is used to construct the second training data, which includes several second training images carrying title elements or subtitle elements. The third preprocessing unit is used to perform feature preprocessing on the second training image to generate a probability map and a threshold map. The second training unit is used to train the DBNet detection network based on a preset loss function to obtain a target text detection model with the position information of the text as the output data. The text recognition unit is used to perform text control recognition on the development page and the design page respectively based on the target text detection model, and obtain the second development recognition result and the second design recognition result respectively. The second comparison unit is used to compare the second development identification result with the second design identification result to obtain the second comparison result.

7. The UI control recognition device according to claim 6, characterized in that, Also includes: The combined recognition unit is used to combine the first comparison result with the second comparison result to determine the position information of the graphic and text combination control.

8. A UI control recognition device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the UI control recognition method according to any one of claims 1-4 according to the instructions in the program code.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the UI control recognition method according to any one of claims 1-4.

Images