Page layout processing method and device, equipment and medium

Abstract

The invention discloses a page layout processing method and device, equipment and a medium. The method comprises the steps of obtaining an image of a development page; inputting the image of the development page into a trained image segmentation model, and performing control segmentation on the image of the development page to obtain a control segmentation graph; acquiring control information of the development page, and matching the control information with the control segmentation graph; and correcting the control position in the control segmentation map according to a matching result, and constructing to obtain a control layout of the development page. On the basis that the image of the development page is segmented to obtain the control segmentation map, the recognized control information is matched with the control segmentation map at the front end, accurate correction of the position of the control of the development page is achieved, high-precision page layout construction is obtained through construction, and the accuracy of page layout processing is improved.

Description

Technical Field

[0001] The present invention relates to the field of research and development technology, is applicable to the financial field, and is applied to UI walkthrough scenarios of bank web page development, and in particular to a page layout processing method, device, equipment and medium. Background Art

[0002] In the design and development process of bank web pages, page layout information is an indispensable part, which affects user experience, development efficiency and the quality of the final product. Therefore, in the UI walkthrough of developing web pages, page layout walkthrough is very important.

[0003] In order to conduct efficient and accurate UI walkthrough, page layout construction is required. The information of the development page can be simply divided into two levels. The first level is based on intuitive visual images, and the second level is based on digital control information. The images are intuitive but the control information cannot be expressed directly. The control front-end information is accurate but the display is not intuitive enough and there is a lot of nesting. It is difficult to achieve high-precision page layout restoration, which affects the accuracy of subsequent UI walkthrough task processing. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a page layout processing method, device, equipment and medium that can be applied to financial technology or other related fields. Its main purpose is to effectively combine the image and control information of the development page to achieve high-precision page layout construction and improve the performance of downstream tasks such as UI inspection.

[0005] The technical solution of the present invention is as follows:

[0006] A first aspect of the present invention provides a method for processing a page layout, comprising:

[0007] Get an image of the development page;

[0008] Inputting the image of the development page into a trained image segmentation model, performing control segmentation on the image of the development page to obtain a control segmentation map;

[0009] Acquire control information of the development page, and match the control information with the control segmentation map;

[0010] The control positions in the control segmentation map are corrected according to the matching results, and the control layout of the development page is constructed.

[0011] In one embodiment, the step of inputting the image of the development page into a trained image segmentation model, performing control segmentation on the image of the development page, and obtaining a control segmentation map includes:

[0012] Inputting the image of the development page into a trained image segmentation model, wherein the image segmentation model includes a backbone network layer, a bottleneck layer, and an output layer connected in sequence;

[0013] Using the backbone network layer to extract features from the image of the development page to obtain a corresponding feature map;

[0014] Using the bottleneck layer to capture multi-scale features in the feature map and perform feature aggregation and upsampling processing;

[0015] The output layer is used to convert the upsampled feature map into a binary segmentation map, in which the pixels of the control area are foreground and the pixels of other areas are background.

[0016] In one embodiment, the bottleneck layer includes a spatial pyramid pooling layer and an upsampling layer, the spatial pyramid pooling layer includes a plurality of hole convolution layers of different sizes, the upsampling layer includes a deconvolution layer, and the method of using the bottleneck layer to capture multi-scale features in the feature map and perform feature aggregation and upsampling processing includes:

[0017] The feature map output by the backbone network layer is input into the spatial pyramid pooling layer, and the feature map is captured by a plurality of hole convolution layers of different sizes with different hole rates to obtain feature maps of different scales;

[0018] Aggregate the feature maps of different scales output by the hole convolution layers of all sizes to obtain the feature aggregation map;

[0019] The feature aggregation map is input into the upsampling layer, and the feature aggregation map is upsampled by the deconvolution layer to restore the feature aggregation map to the original size of the image of the development page.

[0020] In one embodiment, the up-sampled feature map is converted into a binary segmentation map by using the output layer, where pixels in the control area are foreground and pixels in other areas are background, including:

[0021] Linearly combine each channel of the upsampled feature map to generate a single-channel feature map;

[0022] The probability that each pixel in the single-channel feature map is a control area is determined, and the single-channel feature map is converted into a corresponding binary segmentation map according to the probability of each pixel, wherein the pixels in the control area are foregrounds, and the pixels in other areas are backgrounds.

[0023] In one embodiment, the acquiring the control information of the development page and matching the control information with the control segmentation map includes:

[0024] Sending an HTTP request to obtain the HTML document of the development page, and parsing the received HTML document of the development page into a DOM tree;

[0025] Traversing the DOM tree, identifying the control elements in the DOM tree and acquiring the control information of the control elements;

[0026] Determine the position of the circumscribed rectangle of each control graphic in the control segmentation diagram;

[0027] The position of each control element in the control information is matched with the position of each bounding rectangle by intersection and comparison.

[0028] In one embodiment, the correcting the control position in the control segmentation map according to the matching result to construct the control layout of the development page includes:

[0029] Get the intersection and union ratio matching result of the position of each control element and the position of each bounding rectangle;

[0030] According to the intersection-and-union matching result, for each control graphic, a development control element that meets a preset condition is selected from all control elements to obtain a development control element set;

[0031] The control layout of the development page is constructed on the blank image according to the position of each development control element in the development control element set.

[0032] In one embodiment, after correcting the control positions in the control segmentation map according to the matching results and constructing the control layout of the development page, the method further includes:

[0033] Obtaining a design page corresponding to the development page;

[0034] Extracting design control elements from the design page to obtain control information of the design control elements in the design page;

[0035] The control information of the developed control element is compared with the control information of the designed control element to obtain a comparison result.

[0036] A second aspect of the present invention provides a page layout processing device, comprising:

[0037] The acquisition module is used to obtain the image of the development page;

[0038] An image segmentation module, used for inputting the image of the development page into a trained image segmentation model, performing control segmentation on the image of the development page, and obtaining a control segmentation map;

[0039] A matching module, used for obtaining control information of the development page, and matching the control information with the control segmentation map;

[0040] The correction construction module is used to correct the control position in the control segmentation map according to the matching result, and construct the control layout of the development page.

[0041] A third aspect of the present invention provides a computer device, comprising at least one processor; and

[0042] a memory communicatively connected to the at least one processor; wherein,

[0043] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned page layout processing method.

[0044] A fourth aspect of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned page layout processing method.

[0045] Beneficial effects: The present invention discloses a method, device, equipment and medium for processing page layout. Compared with the prior art, the embodiment of the present invention obtains an image of a development page; inputs the image of the development page into a trained image segmentation model, performs control segmentation on the image of the development page, and obtains a control segmentation map; obtains control information of the development page, matches the control information with the control segmentation map; corrects the control positions in the control segmentation map according to the matching results, and constructs a control layout of the development page. Through the trained image segmentation model, the controls on the development page can be automatically and efficiently identified and segmented. On the basis of automatically segmenting and obtaining the control segmentation map, the identified control information is matched with the control segmentation map of the front end to achieve accurate correction of the control positions of the development page, and a high-precision page layout is constructed. Combining the intuitiveness of the image and the accuracy of the control information, a more comprehensive page layout can be constructed, providing richer information for UI walkthroughs, and more accurate UI walkthroughs to ensure that the page layout meets the design specifications. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the scheme in the present invention, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A flow chart of a page layout processing method provided by an embodiment of the present invention;

[0048] Figure 2 A flow chart of step S102 in the page layout processing method provided by an embodiment of the present invention;

[0049] Figure 3 A flow chart of step S103 in the page layout processing method provided by an embodiment of the present invention;

[0050] Figure 4 A flow chart of step S104 in the page layout processing method provided by an embodiment of the present invention;

[0051] Figure 5 A flowchart of UI walkthrough in the page layout processing method provided by an embodiment of the present invention;

[0052] Figure 6 A schematic diagram of functional modules of a page layout processing device provided by an embodiment of the present invention;

[0053] Figure 7 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0055] In the design and development process of bank web pages, page layout information is an indispensable part, which affects user experience, development efficiency and the quality of the final product. Therefore, in the UI walkthrough of developing web pages, page layout walkthrough is very important.

[0056] In order to conduct efficient and accurate UI walkthrough, page layout construction is required. The information of the development page can be simply divided into two levels. The first level is based on intuitive visual images, and the second level is based on digital control information. The images are intuitive but the control information cannot be expressed directly. The control front-end information is accurate but the display is not intuitive enough and there is a lot of nesting. It is difficult to achieve high-precision page layout restoration, which affects the accuracy of subsequent UI walkthrough task processing.

[0057] In order to solve the above problems, the present invention proposes a method for processing page layout. Figure 1 , Figure 1A flowchart of an embodiment of a method for processing page layout provided by the present invention. The system architecture applied to the method for processing page layout provided in this embodiment may include a first terminal device, a second terminal device, a third terminal device, a network and a server. The network is used to provide a medium for a communication link between the first terminal device, the second terminal device, the third terminal device and the server. The network may include various connection types, such as wired and / or wireless communication links, etc.

[0058] The user may use the first terminal device, the second terminal device, and the third terminal device to interact with the server through the network to receive or send messages, etc. Various communication client applications may be installed on the first terminal device, the second terminal device, and the third terminal device, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).

[0059] The first terminal device, the second terminal device, and the third terminal device may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0060] The server may be a server that provides various services, such as a background management server that provides support for the content browsed by the user using the first terminal device, the second terminal device, and the third terminal device (for example only). The background management server may analyze and process the received data such as the user request, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server for a distributed system, or a server combined with a blockchain.

[0061] It should be noted that the page layout processing method provided in the embodiment of the present invention can generally be executed by the first terminal device, the second terminal device or the third terminal device. Correspondingly, the page layout processing device provided in the embodiment of the present invention can also be set in the first terminal device, the second terminal device or the third terminal device.

[0062] Alternatively, the page layout processing method provided by the embodiment of the present invention may also be generally executed by a server. Accordingly, the page layout processing device provided by the embodiment of the present invention may generally be arranged in a server. The page layout processing method provided by the embodiment of the present invention may also be executed by a server or a server cluster that is different from the server and can communicate with the first terminal device, the second terminal device, the third terminal device and / or the server. Accordingly, the page layout processing device provided by the embodiment of the present invention may also be arranged in a server or a server cluster that is different from the server and can communicate with the first terminal device, the second terminal device, the third terminal device and / or the server.

[0063] It should be understood that the numbers of the above terminal devices, networks and servers are only illustrative. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0064] like Figure 1 As shown, the page layout processing method provided in this embodiment specifically includes the following steps:

[0065] S101. Obtain an image of a development page.

[0066] In this embodiment, the development page refers to a web page or application program interface created and developed by the developer according to the corresponding design page or design draft during the software development process. The development page is a user interface (UI) for human-computer interaction, which can be an H5 page, a computer page (i.e., a PC page), an application (APP) page, a mini-program page, etc. The image of the development page is captured by a screenshot tool or an automated testing tool, and the image can clearly, intuitively, and completely display the front-end visual information of the development page, providing a reliable and complete data basis for subsequent page layout processing.

[0067] In one specific implementation, in a banking business scenario, the images of the development page obtained include but are not limited to the following types:

[0068] Login page: the interface where users enter their username and password for identity authentication;

[0069] Account management page: a page where users can view account balances, transaction records, and other information;

[0070] Transfer page: the interface where users conduct fund transfer operations;

[0071] Loan application page: the page where users apply for a loan or view loan details;

[0072] Credit card management page: the page where users manage credit card accounts, view bills and make repayments;

[0073] Investment and financial management page: the interface for users to purchase financial products or conduct investment transactions;

[0074] Customer service page: a page that provides customer support services such as FAQs and online customer service;

[0075] Security settings page: the page where users set or modify security questions, passwords, and other security-related information;

[0076] Personal information page: the page where users update their personal information, such as contact information, address, etc.

[0077] Notification and Messages Page: This page displays account-related notifications, transaction confirmations, and other messages.

[0078] By acquiring an image of the development page and processing the page layout during the development of the web page with interactive controls, the accuracy of page layout construction in bank web page design and development can be improved, ensuring the quality of the final product and user experience.

[0079] S102: Input the image of the development page into a trained image segmentation model, perform control segmentation on the image of the development page, and obtain a control segmentation map.

[0080] In this embodiment, the image of the development page obtained is input into a trained image segmentation model, and the image of the development page is automatically segmented for controls through deep learning or machine learning algorithms to obtain a control segmentation map. That is, the trained image segmentation model can identify and distinguish different controls in the image, such as buttons, text boxes, drop-down menus, etc., based on the front-end visual information of the development page. The specific image segmentation model can be, for example, an FCN (Fully Convolutional Network) model, a DeepLab model, a Mask R-CNN model, a FastFCN model, and the like. The controls in the image are automatically identified by a trained image segmentation model, the need for manual segmentation is reduced, the efficiency of control identification and segmentation is improved, and automatic control segmentation is performed through the intuitive visual information of the development web page, so that preliminary control identification can be achieved, and preliminary results are provided for further analysis and matching of the controls.

[0081] In one specific implementation, in a banking business scenario, for the login page of the bank website, a screenshot of the bank web login page is input into a trained image segmentation model. Of course, the image can also be preprocessed before being input into the model, including resizing, normalizing brightness and contrast, and denoising, etc., to improve the performance of the image segmentation model. The preprocessed login page image is input into the trained image segmentation model, and the model is used to identify controls in the login page image, such as text input boxes, password input boxes, login buttons, etc., to efficiently obtain preliminary page layout information.

[0082] S103: Acquire control information of the development page, and match the control information with the control segmentation map.

[0083] In this embodiment, while obtaining a preliminary control segmentation map based on an intuitive visual image, accurate control information of the development page is also obtained. The specific control information may include control type: indicating the category of the control, such as button (Button), text box (TextBox), drop-down menu (Dropdown), check box (Checkbox), radio button (RadioButton), slider (Slider), etc.; control identifier: including the control ID, name (Name), class (Class), etc., which are the unique identifiers of the control in the HTML document; position information: X and Y coordinates, the position of the control on the page, usually the distance relative to the upper left corner of the page, the width and height of the control, etc.; style attributes: the appearance style of the control, such as color, border, padding, font size, etc.; status information: the current state of the control, such as whether it is disabled (disabled), whether it is selected (checked), whether it is visible (visible), etc. Specifically, relevant information of the control can be obtained from the source code of the development web page or through the API, and then the extracted control information is matched with the control segmentation map output by the image segmentation model. Although the control information is accurate, the display is not intuitive enough and there is a lot of nesting. Therefore, an intuitive control segmentation map is obtained through image segmentation, and then the control information is matched with the control segmentation map to identify the exact position and properties of each control, integrating intuitive visual images with accurate control information, and providing detailed data for subsequent layout correction.

[0084] In one specific implementation, in a banking business scenario, for the login page of a bank website, control information can be extracted from the DOM (document object model) of the bank webpage, including the control type (such as a text input box, a password input box, a login button, etc.), the control ID, class name, attributes, and expected layout coordinates, etc., and the control segmentation map output by the image segmentation model is matched with the control information in the DOM. For example, feature points of each control, such as the four corner points of the bounding box, are extracted from the control segmentation map. These points define the position and size of the control. The matching relationship is determined by calculating the proximity between the feature points and the control coordinates recorded in the DOM, so that the intuitive control segmentation results can be combined with the precise control information to match and obtain accurate information of each control.

[0085] S104: Correct the control positions in the control segmentation map according to the matching results, and construct a control layout of the development page.

[0086] In this embodiment, the positions of each control in the control segmentation map are corrected by combining the matching results between the intuitive front-end image segmentation results and the precise control information. Specifically, the degree of matching between the output of the image segmentation model and the actual control information can be analyzed, and the correction targets corresponding to each control graphic in the control segmentation map (i.e., the precise control information of each control graphic) are determined according to the matching degree. Based on the correction targets, the corresponding control graphics are subjected to correction operations such as boundary alignment, size adjustment, and overlap processing, thereby combining the initially identified control segmentation map with the precise control information to construct a final high-precision page layout, providing richer information for UI walkthroughs, and performing UI walkthroughs more accurately to ensure that the page layout meets the design specifications.

[0087] In one specific implementation, in a banking business scenario, for the login page of a bank website, based on the matching result between the control segmentation diagram and the control information in the login page, the expected position of each control in the control segmentation diagram is determined. If it is found that the control position in the control segmentation diagram deviates from the expected position in the DOM, the position of the control is fine-tuned. For example, if the position of an input box is to the left or to the top than expected, its coordinates are adjusted to match the expected value, thereby constructing a high-precision control layout for the login page, providing an accurate data basis for the control layout verification, control functionality testing, etc. of the login page.

[0088] In the above embodiment, the present invention discloses a method for processing page layout, by obtaining an image of a development page; inputting the image of the development page into a trained image segmentation model, performing control segmentation on the image of the development page, and obtaining a control segmentation map; obtaining control information of the development page, and matching the control information with the control segmentation map; correcting the control positions in the control segmentation map according to the matching results, and constructing a control layout of the development page. Through the trained image segmentation model, the controls on the development page can be automatically and efficiently identified and segmented. On the basis of automatically segmenting and obtaining the control segmentation map, the identified control information is matched with the control segmentation map of the front end to achieve accurate correction of the control positions of the development page, and a high-precision page layout can be constructed. Combining the intuitiveness of the image and the accuracy of the control information, a more comprehensive page layout can be constructed, providing richer information for UI walkthroughs, and more accurately conducting UI walkthroughs to ensure that the page layout meets the design specifications.

[0089] In one embodiment, Figure 2 As shown, step S102 includes:

[0090] S201, inputting the image of the development page into a trained image segmentation model, wherein the image segmentation model includes a backbone network layer, a bottleneck layer, and an output layer connected in sequence;

[0091] S202, using the backbone network layer to extract features from the image of the development page to obtain a corresponding feature map;

[0092] S203, using the bottleneck layer to capture multi-scale features in the feature map and perform feature aggregation and upsampling processing;

[0093] S204, using the output layer to convert the upsampled feature map into a binary segmentation map, where pixels in the control area are foreground and pixels in other areas are background.

[0094] In this embodiment, an image segmentation model is pre-trained for segmentation to obtain a control segmentation map. In the training stage of the model, a data set for model training is first collected. The data set comes from a development web page that requires a UI walkthrough. The rendered images of the development page are collected in the form of a traversal address set, and then the input boxes in the page are labeled in the form of a binary map. Then, an image segmentation model is constructed, which includes a backbone network layer, a bottleneck layer, and an output layer connected in sequence. The backbone network layer is first composed of 1x1 convolution, 3x3 convolution, and 1x1 convolution connected in sequence to form an anti-pyramid structure, and then the activation layer and the short connection layer form basic units. The basic units are connected in sequence to form a block, and finally four blocks constitute the backbone network layer; the bottleneck layer is composed of a spatial pyramid pooling layer (Spatial Pyramid Pooling, SPP) and an upsampling layer. The SPP layer aggregates the output of the backbone network through 5x 5, 7x 7, and 9x 9 hole convolutions and connects it to a 1x1 convolution. The upsampling layer is realized by deconvolution; the output layer is composed of a simple 1x1 convolution layer. After that, the model is trained. Specifically, the public imagenet pre-trained model is first loaded into the backbone network to complete the initialization of the backbone network, and the optimizer is configured as adam, the loss function is CELoss, the training batch is 128, the learning rate is 1e-3, and the model is trained until the loss converges. The corresponding parameters are saved, thereby completing the training of the image segmentation model.

[0095] Based on the above-trained image segmentation model, the image of the development page is input into the pre-trained image segmentation model, and the input image is processed in sequence by using multiple superimposed blocks in the backbone network layer of the model. The output of each block becomes the input of the next block, and the image of the development page is extracted. Finally, the backbone network layer outputs a set of feature maps after multi-level feature extraction. These feature maps contain spatial information and semantic information in the image and will be sent to the subsequent bottleneck layer and output layer for further processing; for the feature map output by the backbone network layer, the bottleneck layer is used to capture the multi-scale features in the feature map, and multi-scale analysis is used to make The model can recognize controls of different sizes, and also perform feature aggregation and upsampling on the captured multi-scale features, which improves the resolution of the feature map, restores the details of the image, and provides a basis for accurate segmentation; for the upsampled feature map output by the bottleneck layer, it is converted into a binary segmentation map through the output layer, and each pixel of the feature map is converted into a foreground or background label, where the pixels in the control area are the foreground and the pixels in other areas are the background, thereby segmenting the control area from the image, and using the generated binary segmentation map as the control segmentation map to clearly distinguish the foreground (control) and background, and the binarization process can reduce the amount of data processed and improve processing efficiency.

[0096] In one embodiment, step S203 includes:

[0097] The feature map output by the backbone network layer is input into the spatial pyramid pooling layer, and the feature map is captured by a plurality of hole convolution layers of different sizes with different hole rates to obtain feature maps of different scales;

[0098] Aggregate the feature maps of different scales output by the hole convolution layers of all sizes to obtain the feature aggregation map;

[0099] The feature aggregation map is input into the upsampling layer, and the feature aggregation map is upsampled by the deconvolution layer to restore the feature aggregation map to the original size of the image of the development page.

[0100] In this embodiment, when the bottleneck layer processes the feature map, it first captures the feature map with different dilation rates through multiple dilated convolution layers of different sizes in the spatial pyramid pooling layer. For example, multiple dilated convolutions of different sizes are applied to the same input feature map, such as 3x3, 5x5, 7x7, etc., and each size of the convolution kernel uses a different dilation rate. rate), each hole convolution of different sizes outputs a feature map, which captures features of different scales and can capture local details and global context information of the image at the same time, improving the model's ability to recognize objects of different sizes; then the feature maps output by the hole convolution of all sizes are aggregated and connected to 1x1 convolution to adjust the dimension of the feature to match the input requirements of the subsequent layers; then the feature map output by the spatial pyramid pooling layer is used as input to the upsampling layer, and the feature aggregation map is upsampled through the deconvolution layer to increase the size of the feature aggregation map until the feature aggregation map is restored to the original size of the image on the development page, thereby reconstructing the detail information of the image in the upsampling process, and the upsampled feature map will be used for the final segmentation task to prepare for obtaining high-resolution segmentation results.

[0101] In one embodiment, step S204 includes:

[0102] Linearly combine each channel of the upsampled feature map to generate a single-channel feature map;

[0103] The probability that each pixel in the single-channel feature map is a control area is determined, and the single-channel feature map is converted into a corresponding binary segmentation map according to the probability of each pixel, wherein the pixels in the control area are foregrounds, and the pixels in other areas are backgrounds.

[0104] In this embodiment, the 1x1 convolution layer in the output layer linearly combines the upsampled multi-channel feature maps to generate a single-channel feature map, and integrates the information of the multi-channel feature maps into a single channel, which is convenient for subsequent processing, reduces the dimension of the data, and simplifies the complexity of the model. After the 1x1 convolution layer, the output of the convolution layer can be compressed between 0 and 1 through the Sigmoid activation function, and the value of each pixel in the single-channel feature map is converted into the probability of belonging to the foreground, that is, the probability of belonging to the control area. The single-channel feature map is converted into a corresponding binary segmentation map according to the probability of each pixel. Specifically, all pixels greater than or equal to the threshold can be marked as foreground (usually marked as 1), and all pixels less than the threshold can be marked as background (usually marked as 0). The threshold can be a fixed value such as 0.5 or a dynamic value, which can be flexibly adjusted according to actual needs, and this embodiment is not limited to this. Through probabilistic judgment, the model can more accurately segment the control area, convert continuous probability maps into binary maps, and simplify the subsequent processing of segmentation results, thereby achieving accurate segmentation of controls in development page images and providing high-quality segmentation results for subsequent control identification, layout construction, and automated testing.

[0105] In one embodiment, Figure 3 As shown, step S103 includes:

[0106] S301, sending an HTTP request to obtain the HTML document of the development page, and parsing the received HTML document of the development page into a DOM tree;

[0107] S302, traversing the DOM tree, identifying control elements in the DOM tree and acquiring control information of the control elements;

[0108] S303, determining the position of the circumscribed rectangle of each control graphic in the control segmentation diagram;

[0109] S304: Perform an intersection and comparison match between the position of each control element in the control information and the position of each circumscribed rectangle.

[0110] In this embodiment, when acquiring the control information of the development page, the acquisition code of the page information can be generated by traversing the front-end page elements through JavaScript, and injected into the corresponding development page. When the browser opens the relevant development page, the acquisition of the development information can be completed. Specifically, the browser sends an HTTP request to the server to obtain the HTML document of the development page. The server responds to the request and returns the content of the HTML document. After the browser receives the HTML document, it starts to parse the document and converts it into a DOM (Document Object Model) tree. The DOM tree is a memory representation of the HTML document, which structures the document into a tree structure composed of nodes (elements, attributes, text, etc.). Based on the JavaScript code injected into the development page, the DOM tree is traversed, the control elements in the DOM tree are identified, and the control information of the control elements is obtained. For example, the DOM API provided by JavaScript, such as document.getElementById, document.querySelector, etc., can be used to query and traverse the page elements, obtain the properties, style, position, size and other information of the control, so as to obtain the accurate control information of the control element in all elements.

[0111] On the basis of obtaining accurate control information, the position of the bounding rectangle of each control graphic in the control segmentation map is also determined. For example, the boundingRect method in the opencv library can be used to obtain the position of the bounding rectangle of each control graphic in the binary control segmentation map. Image processing technology (such as connected domain analysis or contour detection) can also be used to determine the bounding rectangle of each control graphic and calculate the position and size of the bounding rectangle. Each control graphic provides accurate position and size information. The position of each control element in the traversed control information is matched with the position of the bounding rectangle of each control graphic obtained by segmentation, that is, the intersection over union (IoU) between the position of each bounding rectangle and the position of each control element in the control information is calculated. Through IoU matching, the control graphics in the segmentation map can be accurately matched with the control elements in the DOM tree, thereby matching the exact position and attributes of each control graphic in the control segmentation map, providing a basis for subsequent correction and verification.

[0112] In one embodiment, Figure 4 As shown, step S104 includes:

[0113] S401, obtaining the intersection-and-union matching result of the position of each control element and the position of each circumscribed rectangle;

[0114] S402, selecting a development control element that meets a preset condition for each control graphic from all control elements according to the intersection-and-union matching result, to obtain a development control element set;

[0115] S403: constructing a control layout of the development page on a blank image according to the position of each development control element in the development control element set.

[0116] In this embodiment, for each segmented control graphic, the intersection over union (IoU) of its circumscribed rectangle position and the position of each control element in the DOM is calculated, and based on the IoU matching result, a development control element that meets the preset conditions is screened for each control graphic from all the control elements, and a quantitative matching accuracy index is provided through the IoU matching result, which provides a basis for the subsequent screening of control elements. Specifically, from all the control elements, the element with an intersection over union greater than a preset threshold (for example, 0.6) and the largest IoU is found for each control graphic in turn as a development control element, ensuring that only controls with high matching accuracy are selected, reducing the number of control elements that are incorrectly matched or inaccurately matched, and the found development control elements are saved in the development control element set until the corresponding development control elements are matched for all control graphics. When constructing the control layout of the development page, create a blank image of the same size as the original development page image, and draw the circumscribed rectangle or mark of each development control element on the blank image based on the position of each development control element in the development control element set. Combine the initially identified control segmentation map with the precise control information to construct the final high-precision page layout, provide richer information for UI walkthrough, conduct UI walkthrough more accurately, and ensure that the page layout meets the design specifications.

[0117] In one embodiment, Figure 5 As shown, after step S104, the method further includes:

[0118] S501, obtaining a design page corresponding to the development page;

[0119] S502, extracting design control elements from the design page to obtain control information of the design control elements in the design page;

[0120] S503: Compare the control information of the developed control element with the control information of the designed control element to obtain a comparison result.

[0121] In this embodiment, after constructing a high-precision development page control layout, an efficient UI walkthrough can be performed. First, the design page corresponding to the development page is obtained, the design page file is parsed, the design control elements in the design page are extracted, and the position, size, style and other control information of the design control elements are obtained using the API or plug-in provided by the design software, so as to accurately obtain the detailed information of the controls in the design page. After that, based on the constructed control layout of the development page, the control information of the development control elements is compared with the control information of the design control elements, and the position, size, style, etc. of each control are checked to see if they are consistent. Automated tools or scripts can be used to compare properties to obtain comparison results. Since the design page elements and the development page elements are the same under normal circumstances, if there are differences, the differences can be recorded based on the comparison results, and the differences between the development page and the design page can be quickly discovered and adjusted to ensure that the development results meet the design specifications and expected effects.

[0122] It should be noted that there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.

[0123] Further references Figure 6 , as a response to the above Figure 1 The present invention provides an embodiment of a device for processing page layout, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0124] like Figure 6 As shown, the page layout processing device 60 described in this embodiment includes:

[0125] The acquisition module 601 is used to acquire the image of the development page;

[0126] An image segmentation module 602 is used to input the image of the development page into a trained image segmentation model, perform control segmentation on the image of the development page, and obtain a control segmentation map;

[0127] A matching module 603 is used to obtain the control information of the development page and match the control information with the control segmentation map;

[0128] The correction construction module 604 is used to correct the control positions in the control segmentation map according to the matching results, and construct the control layout of the development page.

[0129] The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions, which are more suitable for describing the processing and execution process of page layout than programs. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0130] In one embodiment, the image segmentation module 602 includes:

[0131] An input unit, used for inputting the image of the development page into a trained image segmentation model, wherein the image segmentation model comprises a backbone network layer, a bottleneck layer and an output layer connected in sequence;

[0132] A feature extraction unit, used to extract features from the image of the development page using the backbone network layer to obtain a corresponding feature map;

[0133] A feature processing unit, used to capture multi-scale features in the feature map using the bottleneck layer and perform feature aggregation and upsampling processing;

[0134] The conversion output unit is used to convert the upsampled feature map into a binary segmentation map using the output layer, wherein the pixels of the control area in the binary segmentation map are foreground and the pixels of other areas are background.

[0135] In one embodiment, the bottleneck layer includes a spatial pyramid pooling layer and an upsampling layer, the spatial pyramid pooling layer includes a plurality of hole convolution layers of different sizes, the upsampling layer includes a deconvolution layer, and the feature processing unit includes:

[0136] A multi-scale feature processing unit, used to input the feature map output by the backbone network layer into the spatial pyramid pooling layer, and to capture the feature map with different hole rates through a plurality of hole convolution layers of different sizes to obtain feature maps of different scales;

[0137] The feature aggregation unit is used to aggregate the feature maps of different scales output by the dilated convolutional layers of all sizes to obtain a feature aggregation map;

[0138] An upsampling unit is used to input the feature aggregation map into the upsampling layer, perform an upsampling operation on the feature aggregation map through the deconvolution layer, and restore the feature aggregation map to the original size of the image of the development page.

[0139] In one embodiment, the conversion output unit includes:

[0140] A channel combination unit is used to linearly combine each channel of the upsampled feature map to generate a single-channel feature map;

[0141] The binary image conversion unit is used to determine the probability that each pixel in the single-channel feature image is a control area, and convert the single-channel feature image into a corresponding binary segmentation image according to the probability of each pixel, wherein the pixels in the control area are foregrounds, and the pixels in other areas are backgrounds.

[0142] In one embodiment, the matching module 603 includes:

[0143] A parsing unit, used for sending an HTTP request to obtain the HTML document of the development page, and parsing the received HTML document of the development page into a DOM tree;

[0144] An element traversal identification unit, used to traverse the elements of the DOM tree, identify the control elements in the DOM tree and obtain the control information of the control elements;

[0145] A graphic position determination unit, used to determine the position of the circumscribed rectangle of each control graphic in the control segmentation diagram;

[0146] The position matching unit is used to perform intersection and union matching on the position of each control element in the control information and the position of each circumscribed rectangle.

[0147] In one embodiment, the correction construction module 604 includes:

[0148] A result acquisition unit, used to acquire the intersection-and-union ratio matching result of the position of each control element and the position of each circumscribed rectangle;

[0149] An element screening unit, used for screening development control elements that meet preset conditions for each control graphic from all control elements according to the intersection-and-union matching result, to obtain a development control element set;

[0150] The correction construction unit is used to construct the control layout of the development page on the blank image according to the position of each development control element in the development control element set.

[0151] In one embodiment, the device 60 further includes:

[0152] The inspection module is used to obtain the design page corresponding to the development page; and to extract the design control elements from the design page to obtain the control information of the design control elements in the design page; and to compare the control information of the development control elements with the control information of the design control elements to obtain a comparison result.

[0153] In the above embodiment, the present invention discloses a page layout processing device, which obtains an image of a development page; inputs the image of the development page into a trained image segmentation model, performs control segmentation on the image of the development page, and obtains a control segmentation map; obtains control information of the development page, matches the control information with the control segmentation map; corrects the control positions in the control segmentation map according to the matching results, and constructs a control layout of the development page. Through the trained image segmentation model, the controls on the development page can be automatically and efficiently identified and segmented. On the basis of automatically segmenting and obtaining the control segmentation map, the identified control information is matched with the control segmentation map of the front end to achieve accurate correction of the control positions of the development page, and a high-precision page layout is constructed. Combining the intuitiveness of the image and the accuracy of the control information, a more comprehensive page layout can be constructed, providing richer information for UI walkthroughs, and more accurate UI walkthroughs to ensure that the page layout meets the design specifications.

[0154] Another embodiment of the present invention provides a computer device, such as Figure 7 As shown, the computer device 70 includes:

[0155] One or more processors 701 and memory 702, Figure 7 A processor 701 is used as an example for description. The processor 701 and the memory 702 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0156] Processor 701 is used to complete various control logics of computer device 70, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component or any combination of these components. In addition, processor 701 can also be any traditional processor, microprocessor or state machine. Processor 701 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0157] The memory 702 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the page layout processing method in the embodiment of the present invention. The processor 701 executes various functional applications and data processing of the computer device 70 by running the non-volatile software programs, instructions and units stored in the memory 702, that is, implements the page layout processing method in the above method embodiment.

[0158] The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device 70, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 702 may optionally include a memory remotely arranged relative to the processor 701, and these remote memories may be connected to the computer device 70 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. One or more units are stored in the memory 702, and when executed by one or more processors 701, the steps of the page layout processing method in any of the above-mentioned method embodiments are executed.

[0159] In the above embodiment, the present invention discloses a computer device, which obtains an image of a development page; inputs the image of the development page into a trained image segmentation model, performs control segmentation on the image of the development page, and obtains a control segmentation map; obtains control information of the development page, matches the control information with the control segmentation map; corrects the control positions in the control segmentation map according to the matching results, and constructs a control layout of the development page. Through the trained image segmentation model, the controls on the development page can be automatically and efficiently identified and segmented. On the basis of automatically segmenting and obtaining the control segmentation map, the identified control information is matched with the control segmentation map of the front end to achieve accurate correction of the control positions of the development page, and a high-precision page layout is constructed. Combining the intuitiveness of the image and the accuracy of the control information, a more comprehensive page layout can be constructed, providing richer information for UI walkthroughs, and more accurate UI walkthroughs to ensure that the page layout meets the design specifications.

[0160] An embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the steps of the page layout processing method in any of the above method embodiments are executed.

[0161] In the above embodiment, the present invention discloses a non-volatile computer-readable storage medium, which obtains an image of a development page; inputs the image of the development page into a trained image segmentation model, performs control segmentation on the image of the development page, and obtains a control segmentation map; obtains control information of the development page, matches the control information with the control segmentation map; corrects the control position in the control segmentation map according to the matching result, and constructs a control layout of the development page. Through the trained image segmentation model, the controls on the development page can be automatically and efficiently identified and segmented. On the basis of automatically segmenting and obtaining the control segmentation map, the identified control information is matched with the control segmentation map of the front end to achieve accurate correction of the control position of the development page, and a high-precision page layout is constructed. Combining the intuitiveness of the image and the accuracy of the control information, a more comprehensive page layout can be constructed, providing richer information for UI walkthroughs, and more accurately conducting UI walkthroughs to ensure that the page layout meets the design specifications.

[0162] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0163] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0164] In summary, the present invention discloses a method, device, equipment and medium for processing page layout, the method comprising: obtaining an image of a development page; inputting the image of the development page into a trained image segmentation model, performing control segmentation on the image of the development page, and obtaining a control segmentation map; obtaining control information of the development page, and matching the control information with the control segmentation map; correcting the control positions in the control segmentation map according to the matching results, and constructing a control layout for the development page. Through the trained image segmentation model, the controls on the development page can be automatically and efficiently identified and segmented. On the basis of automatically segmenting and obtaining the control segmentation map, the identified control information is matched with the control segmentation map of the front end to achieve accurate correction of the control positions of the development page, and a high-precision page layout can be constructed. Combining the intuitiveness of the image and the accuracy of the control information, a more comprehensive page layout can be constructed, providing richer information for UI walkthroughs, and more accurately conducting UI walkthroughs to ensure that the page layout meets the design specifications.

[0165] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0166] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for processing page layout, characterized in that: include: Get an image of the development page; Inputting the image of the development page into a trained image segmentation model, performing control segmentation on the image of the development page to obtain a control segmentation map; Acquire control information of the development page, and match the control information with the control segmentation map; The control positions in the control segmentation map are corrected according to the matching results, and the control layout of the development page is constructed.

2. The page layout processing method according to claim 1, characterized in that: The step of inputting the image of the development page into a trained image segmentation model, performing control segmentation on the image of the development page, and obtaining a control segmentation map includes: Inputting the image of the development page into a trained image segmentation model, wherein the image segmentation model includes a backbone network layer, a bottleneck layer, and an output layer connected in sequence; Using the backbone network layer to extract features from the image of the development page to obtain a corresponding feature map; Using the bottleneck layer to capture multi-scale features in the feature map and perform feature aggregation and upsampling processing; The output layer is used to convert the upsampled feature map into a binary segmentation map, in which the pixels of the control area are foreground and the pixels of other areas are background.

3. The page layout processing method according to claim 2, characterized in that: The bottleneck layer includes a spatial pyramid pooling layer and an upsampling layer, the spatial pyramid pooling layer includes a plurality of hole convolution layers of different sizes, the upsampling layer includes a deconvolution layer, and the bottleneck layer is used to capture multi-scale features in the feature map and perform feature aggregation and upsampling processing, including: The feature map output by the backbone network layer is input into the spatial pyramid pooling layer, and the feature map is captured by a plurality of hole convolution layers of different sizes with different hole rates to obtain feature maps of different scales; Aggregate the feature maps of different scales output by the hole convolution layers of all sizes to obtain the feature aggregation map; The feature aggregation map is input into the upsampling layer, and the feature aggregation map is upsampled by the deconvolution layer to restore the feature aggregation map to the original size of the image of the development page.

4. The page layout processing method according to claim 2, characterized in that: The up-sampled feature map is converted into a binary segmentation map by using the output layer, wherein the pixels of the control area in the binary segmentation map are foreground and the pixels of other areas are background, including: Linearly combine each channel of the upsampled feature map to generate a single-channel feature map; The probability that each pixel in the single-channel feature map is a control area is determined, and the single-channel feature map is converted into a corresponding binary segmentation map according to the probability of each pixel, wherein the pixels in the control area are foregrounds, and the pixels in other areas are backgrounds.

5. The page layout processing method according to claim 1, characterized in that: The acquiring the control information of the development page and matching the control information with the control segmentation map includes: Sending an HTTP request to obtain the HTML document of the development page, and parsing the received HTML document of the development page into a DOM tree; Traversing the DOM tree, identifying the control elements in the DOM tree and acquiring the control information of the control elements; Determine the position of the circumscribed rectangle of each control graphic in the control segmentation diagram; The position of each control element in the control information is matched with the position of each bounding rectangle by intersection and comparison.

6. The page layout processing method according to claim 5, characterized in that: The step of correcting the control position in the control segmentation map according to the matching result to construct a control layout of the development page includes: Get the intersection and union ratio matching result of the position of each control element and the position of each bounding rectangle; According to the intersection-and-union matching result, for each control graphic, a development control element that meets a preset condition is selected from all control elements to obtain a development control element set; The control layout of the development page is constructed on the blank image according to the position of each development control element in the development control element set.

7. The page layout processing method according to claim 6, characterized in that: After correcting the control position in the control segmentation map according to the matching result and constructing the control layout of the development page, the method further includes: Obtaining a design page corresponding to the development page; Extracting design control elements from the design page to obtain control information of the design control elements in the design page; The control information of the developed control element is compared with the control information of the designed control element to obtain a comparison result.

8. A page layout processing device, characterized in that: include: The acquisition module is used to obtain the image of the development page; An image segmentation module, used for inputting the image of the development page into a trained image segmentation model, performing control segmentation on the image of the development page, and obtaining a control segmentation map; A matching module, used for obtaining control information of the development page, and matching the control information with the control segmentation map; The correction construction module is used to correct the control position in the control segmentation map according to the matching result, and construct the control layout of the development page.

9. A computer device, characterized in that: comprising at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the page layout processing method described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the page layout processing method according to any one of claims 1 to 7.

Images