UI control association walking method, device and equipment and storage medium

Abstract

The invention discloses a UI control association walkthrough method, device and equipment and a storage medium, element information corresponding to all controls contained in a front-end page is captured, the controls are classified according to preset labels, a control set is formed, duplicate removal of the control set is further carried out based on the area of each control in the control set, and the user experience is improved. And finally, associating the controls in pairs according to the coordinates and a preset priority, and determining a problem control existing in an associated control set according to a preset control comparison rule, so that condition comparison among elements of the multiple controls is realized, more accurate spacing walk-through is provided, the manual walk-through cost is reduced, and the walk-through efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, equipment and storage medium for checking associations of UI controls. Background Art

[0002] In today's digital age, front-end development has become an indispensable part of software development. As users' requirements for the aesthetics and functionality of web and application interfaces (UI) continue to increase, quality control of front-end development has become particularly important. Among them, the spacing check between page elements is one of the key factors to ensure that the user interface is neat, coordinated, and easy to use.

[0003] There are various forms of control walkthroughs on development pages. Intuitive walkthrough conditions only require comparison of a single control. There is currently a lack of specific solutions for walkthrough conditions such as spacing that require the association of multiple controls. Summary of the invention

[0004] The present application provides a UI control association inspection method, device, equipment and storage medium, which realizes conditional comparison between elements of multiple controls, provides more accurate spacing inspection, reduces manual inspection costs, and improves inspection efficiency.

[0005] In view of this, the first aspect of the present application provides a UI control association walkthrough method, the method comprising:

[0006] Step S1, capturing element information corresponding to all controls on the target page;

[0007] Step S2: classify the element information according to preset labels to form a plurality of control sets corresponding to the preset labels;

[0008] Step S3, based on the area of ​​each control in the control set, performing deduplication processing on each control set respectively;

[0009] Step S4: associate the controls included in the control set after deduplication with each other according to preset priorities and coordinates to form a plurality of associated control sets corresponding to the preset labels;

[0010] Step S5: Based on the preset control comparison rules, the associated control set is compared according to the preset tags to determine the problematic control.

[0011] Optionally, the step S3 is specifically:

[0012] Step S31, calculating the area of ​​each control in a plurality of control sets;

[0013] Step S32, sorting the controls in the control set in descending order of area;

[0014] Step S33, constructing a relational storage array of the control set, wherein the relational storage array includes a front control and a rear control group;

[0015] Step S34, assigning the ith control to the front control according to the sorting traversal;

[0016] Step S35, sequentially calculating the IOU intersection-over-union ratios between the area of ​​the ith control and the areas of the remaining controls after the ith control;

[0017] Step S36: when the IOU intersection-over-union ratio is greater than a preset threshold, assigning the corresponding control after the i-th control to the post-control group;

[0018] Step S37, according to the number of controls included in the post-control group, take out the representative controls after duplicate removal, and obtain the control set after duplicate removal including a number of the representative controls.

[0019] Optionally, the step S37 is specifically:

[0020] When the rear control group contains more than two controls, the control corresponding to the second position in the rear control group is taken out as the representative control, otherwise the control corresponding to the first position in the rear control group is taken out as the representative control.

[0021] Optionally, the step S4 is specifically:

[0022] Step S41, sorting the plurality of representative controls included in the control set in descending order according to the vertical coordinates;

[0023] Step S42: Create an associated control set;

[0024] Step S43, traversing the representative controls included in the control set in order, and adding the representative control combinations associated with each other to the associated control set according to the horizontal coordinates and the preset priority differences.

[0025] Optionally, the step S43 is specifically:

[0026] Step S431, traversing the representative controls included in the control set in order, and adding the i-th representative control to the associated control set;

[0027] Step S432, traverse the remaining representative controls after the ith representative control, and compare the horizontal coordinate difference between the target position of the ith representative control and the remaining representative controls after the ith representative control and the preset priority difference on the front page in turn;

[0028] Step S433: If there is a remaining representative control whose horizontal coordinate difference and the preset priority difference with the i-th representative control are both lower than the corresponding preset threshold, then the corresponding remaining representative control is added to the associated control set as an associated control of the i-th representative control; otherwise, set i=i+1 and return to execute step S431.

[0029] Optionally, the step S431 further includes:

[0030] If the i-th representative component already exists in the associated control set, then i=i+1 and the step S431 is executed again.

[0031] Optionally, after step S5, the following steps are further included:

[0032] Step S6: Pack the information of several problematic controls and send it to the target user for warning.

[0033] A second aspect of the present application provides a UI control association walkthrough device, the device comprising:

[0034] A capture unit is used to capture element information corresponding to all controls on the target page;

[0035] A classification unit, used to classify the element information according to preset labels to form a plurality of control sets corresponding to the preset labels;

[0036] A deduplication unit, used for performing deduplication processing on each control set based on the area of ​​each control in the control set;

[0037] An associating unit, used to associate controls included in the control set after deduplication with each other according to preset priorities and coordinates, to form a plurality of associated control sets corresponding to the preset tags;

[0038] The comparison unit is used to compare the associated control set according to the preset tags based on the preset control comparison rules to determine the problematic control.

[0039] Optionally, the deduplication unit is specifically used for:

[0040] Calculating the area of ​​each control in the plurality of control sets;

[0041] Sort the controls in the control set in descending order of area;

[0042] Constructing a relational storage array of the control set, wherein the relational storage array includes a front control and a rear control group;

[0043] According to the sorting traversal, the i-th control is assigned to the front control;

[0044] Calculate the IOU intersection-over-union ratios between the area of ​​the i-th control and the areas of the remaining controls after the i-th control in sequence;

[0045] When the IOU intersection-over-union ratio is greater than a preset threshold, the corresponding control after the i-th control is assigned to the post-control group;

[0046] According to the number of controls included in the post-control group, the representative controls after duplication are taken out, and the control set after duplication is obtained includes a plurality of the representative controls.

[0047] Optionally, taking out the representative controls after deduplication according to the number of controls included in the post-control group, and obtaining the control set after deduplication including a number of the representative controls is specifically:

[0048] When the rear control group contains more than two controls, the control corresponding to the second position in the rear control group is taken out as the representative control, otherwise the control corresponding to the first position in the rear control group is taken out as the representative control.

[0049] Optionally, the association unit is specifically used to:

[0050] Sort the plurality of representative controls included in the control set in descending order according to the vertical coordinates;

[0051] Create a collection of associated controls;

[0052] The representative controls included in the control set are traversed in order, and according to the horizontal coordinates and the preset priority differences, the representative control combinations associated with each other are added to the associated control set.

[0053] Optionally, the step of sequentially traversing the representative controls included in the control set and adding the representative control combinations associated with each other to the associated control set according to the horizontal coordinates and preset priority differences is specifically as follows:

[0054] Traversing the representative controls included in the control set in order, and adding the i-th representative control to the associated control set;

[0055] Traversing the remaining representative controls after the ith representative control, and comparing the horizontal coordinate differences of the target positions of the ith representative control and the remaining representative controls after the ith representative control and the preset priority differences on the front page in turn;

[0056] If there is a remaining representative control whose horizontal coordinate difference and preset priority difference with the i-th representative control are both lower than the corresponding preset threshold, then the corresponding remaining representative control will be added to the associated control set as the associated control of the i-th representative control, otherwise after i=i+1, return to traverse the representative controls contained in the control set.

[0057] Optionally, the association unit is further used for:

[0058] If the i-th representative component already exists in the associated control set, then after i=i+1, the representative controls included in the control set are traversed again.

[0059] Optionally, it also includes:

[0060] The alarm unit is used to package the information of several problematic controls and send it to the target user for alarm.

[0061] A third aspect of the present application provides a UI control association walkthrough device, the device comprising a processor and a memory:

[0062] The memory is used to store program code and transmit the program code to the processor;

[0063] The processor is used to execute the steps of the method for UI control association walkthrough as described in the first aspect according to the instructions in the program code.

[0064] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the method described in the first aspect above.

[0065] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0066] In the present application, a UI control association inspection method, apparatus, device and storage medium are provided. The method captures the element information corresponding to all controls contained in the front-end page, classifies the controls according to preset labels, forms a control set, further deduplicates the control set based on the area of ​​each control in the control set, and finally associates the controls in pairs according to the coordinates and preset priorities. The problem controls in the associated control set are determined according to the preset control comparison rules, thereby realizing conditional comparison between elements of multiple controls, providing more accurate spacing inspection, reducing the cost of manual inspection, and improving inspection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is a method flow chart of the UI control association walkthrough method in an embodiment of the present application;

[0068] Figure 2 This is a structural diagram of a UI control association walkthrough device in an embodiment of the present application;

[0069] Figure 3 This is a structural diagram of the UI control association walkthrough device in an embodiment of the present application. DETAILED DESCRIPTION

[0070] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0071] The present application designs a UI control association inspection method, device, equipment and storage medium, which realizes the conditional comparison between elements of multiple controls, provides more accurate spacing inspection, reduces manual inspection costs, and improves inspection efficiency.

[0072] For easier understanding, see Figure 1 , Figure 1 : is a method flow chart of the UI control association walkthrough method in the embodiment of the present application, such as Figure 1 As shown, specifically:

[0073] Step S1, capturing element information corresponding to all controls on the target page;

[0074] It should be noted that, for a target page that needs to be walked through, code can be injected into the target page to traverse the target page to capture element information corresponding to all controls.

[0075] Step S2: classify the element information according to preset labels to form a plurality of control sets corresponding to the preset labels;

[0076] It should be noted that the controls are classified according to preset tags in the element information corresponding to the captured controls to form several control sets.

[0077] If the control label is unclear or the control element is difficult to identify, you can use CSS selectors or other query methods to perform more precise queries based on other features of the control (such as position, style, content, etc.);

[0078] Computer vision technology can also be used to analyze page screenshots to identify control elements that are difficult to identify using traditional methods;

[0079] You can also train a machine learning model to identify and classify controls. The model can learn the characteristics of controls from the labeled data and gradually improve the recognition accuracy.

[0080] You can also build or optimize a rules engine that can identify and classify controls based on a complex set of rules and conditions.

[0081] Furthermore, controls that are difficult to identify are used as feedback to continuously iterate and optimize the recognition algorithms and rules.

[0082] Step S3, based on the area of ​​each control in the control set, performing deduplication processing on each control set respectively;

[0083] Specifically:

[0084] Step S31, calculating the area of ​​each control in a plurality of control sets;

[0085] Step S32, sorting the controls in the control set in descending order of area;

[0086] Step S33, constructing a relational storage array of a control set, wherein the relational storage array includes a front control and a rear control group;

[0087] Step S34, traverse in order, assign the i-th control to the preceding control;

[0088] Step S35, sequentially calculating the IOU intersection-over-union ratios between the area of ​​the ith control and the areas of the remaining controls after the ith control;

[0089] Step S36: When the IOU intersection-over-union ratio is greater than a preset threshold, the corresponding controls after the i-th control are assigned to the post-control group;

[0090] Step S37, according to the number of controls included in the post-control group, take out the representative controls after duplicate removal, and obtain a control set after duplicate removal including a number of representative controls;

[0091] Wherein, step S37 is specifically as follows:

[0092] When the rear control group contains more than two controls, the control corresponding to the second position in the rear control group is taken out as the representative control, otherwise the control corresponding to the first position in the rear control group is taken out as the representative control.

[0093] It should be noted that the controls in each control set are calculated and sorted according to their area. Then a relationship storage array is constructed to store the association relationship between controls. Duplicate controls are identified by calculating the IOU value, and duplicates are removed according to the preset threshold, and finally the representative control in each set is determined.

[0094] IOU (Intersection over Union) is an indicator that measures the degree of overlap between two spatial regions and is widely used in image processing, object detection, and machine vision. In the UI control association walkthrough method, IOU is used in the deduplication process to help identify and merge different elements representing the same control.

[0095] The calculation method of IOU intersection-and-union ratio is:

[0096] Given two spatial regions A and B, their intersection region is A∩B, and their union region is A∪B. The IOU intersection-union ratio can be calculated using the following formula:

[0097]

[0098] Among them, |A∩B| represents the area of ​​the overlapping part of regions A and B, and |A∪B| represents the total area covered by A and B.

[0099] When calculating the IOU of a UI control, the control is usually considered to be a rectangular area whose area can be calculated by the width and height of the control. If the control has an irregular shape, more complex geometric algorithms may be needed to determine the area of ​​the intersection and union.

[0100] By calculating the IOU value between controls, we can identify controls that are visually overlapping or very close to each other. These controls may be duplications caused by different HTML elements or style levels.

[0101] When the IOU value exceeds a certain preset threshold (e.g., 0.5 or higher), the two controls can be considered to be similar or different manifestations of the same entity, and thus they are merged in the deduplication process.

[0102] After deduplication, a representative control needs to be selected from the control group with a high IOU value. Usually, controls with a larger area or specific features are selected as representatives for use in subsequent association and comparison processes.

[0103] By removing duplicates, the number of controls that need to be processed in the subsequent association steps can be reduced, thereby improving the efficiency and accuracy of the association process.

[0104] Step S4: associate controls included in the control set after deduplication with each other according to preset priorities and coordinates to form a plurality of associated control sets corresponding to preset labels;

[0105] Specifically:

[0106] Step S41, sorting a number of representative controls included in the control set in descending order according to the vertical coordinates;

[0107] Step S42: Create an associated control set;

[0108] Step S43: traverse the representative controls included in the control set in order, and add the representative control combinations that are associated with each other to the associated control set according to the horizontal coordinates and the preset priority differences.

[0109] Further, step S43 is specifically as follows:

[0110] Step S431, traverse the representative controls included in the control set in order, and add the i-th representative control to the associated control set;

[0111] Step S432, traverse the remaining representative controls after the ith representative control, and compare the horizontal coordinate difference between the target positions of the ith representative control and the remaining representative controls after the ith representative control and the preset priority difference on the front page in turn;

[0112] Step S433: If there is a remaining representative control whose horizontal coordinate difference and preset priority difference with the i-th representative control are both lower than the corresponding preset threshold, then the corresponding remaining representative control will be added to the associated control set as the associated control of the i-th representative control; otherwise, set i=i+1 and return to execute step S431.

[0113] Furthermore, step S431 further includes:

[0114] If the i-th representative component already exists in the associated control set, then i=i+1 and step S431 is executed again.

[0115] It should be noted that the representative controls after deduplication are sorted in descending order according to the ordinate (y coordinate). Then a set of associated controls is created, and two controls are associated according to the abscissa and the preset priority difference. During the association process, if the horizontal distance and priority difference of two controls are both lower than the preset threshold, they are considered to be associated and added to the associated control set.

[0116] If the horizontal coordinate difference and preset priority difference of two controls are both lower than the preset value, this usually means:

[0117] The two controls are close to each other in the horizontal direction. A small difference in the horizontal axis indicates that they are adjacent or closely arranged on the page, which may be an intentional arrangement based on design to keep the interface neat and logical;

[0118] Controls may be visually continuous or belong to the same group of functionally related elements. Smaller horizontal axis differences help maintain visual fluidity and consistency;

[0119] The preset priority is usually set according to the importance or functional ranking of the control in the page. If the difference between the priorities of two controls is small, it indicates that they may be related in function or belong to the same functional module.

[0120] The orderly arrangement and priority ranking of controls help maintain the consistency of the layout. When controls meet the conditions that the horizontal coordinate difference and priority difference are lower than the preset value, they may follow a certain design specification or layout pattern.

[0121] Step S5: Based on the preset control comparison rules, the associated control set is compared according to the preset tags to determine the problematic control.

[0122] It should be noted that the formed associated control set is compared according to the preset control comparison rules. These rules may involve the spacing and alignment between controls. After the comparison, it is determined which controls have problems and do not meet the design specifications.

[0123] Furthermore, after step S5, the following steps are further included:

[0124] Step S6: Pack the information of several problematic controls and send it to the target user for warning.

[0125] It should be noted that the information of the problematic control is packaged and sent to the target user or development team for alerting. This is usually done by generating a report or directly highlighting the problematic control on the user interface.

[0126] See also Figure 2 , Figure 2 Schematic diagram of the structure of the UI control association inspection device in the embodiment of the present application. Figure 2 As shown, specifically:

[0127] Capturing unit 201, used to capture element information corresponding to all controls on the target page;

[0128] It should be noted that, for a target page that needs to be walked through, code can be injected into the target page to traverse the target page to capture element information corresponding to all controls.

[0129] The classification unit 202 is used to classify the element information according to the preset labels to form a plurality of control sets corresponding to the preset labels;

[0130] It should be noted that the controls are classified according to preset tags in the element information corresponding to the captured controls to form several control sets.

[0131] If the control label is unclear or the control element is difficult to identify, you can use CSS selectors or other query methods to perform more precise queries based on other features of the control (such as position, style, content, etc.);

[0132] Computer vision technology can also be used to analyze page screenshots to identify control elements that are difficult to identify using traditional methods;

[0133] You can also train a machine learning model to identify and classify controls. The model can learn the characteristics of controls from the labeled data and gradually improve the recognition accuracy.

[0134] You can also build or optimize a rules engine that can identify and classify controls based on a complex set of rules and conditions.

[0135] Furthermore, controls that are difficult to identify are used as feedback to continuously iterate and optimize the recognition algorithms and rules.

[0136] A deduplication unit 203, configured to perform deduplication processing on each control set based on the area of ​​each control in the control set;

[0137] Specifically:

[0138] Calculate the area of ​​each control in a set of controls;

[0139] Sort the controls in the control collection in descending order of area;

[0140] Construct a relational storage array of a control set, wherein the relational storage array includes a front control and a rear control group;

[0141] According to the sorting traversal, assign the i-th control to the front control;

[0142] Calculate the IOU intersection-and-union ratio between the area of ​​the i-th control and the areas of the remaining controls after the i-th control in sequence;

[0143] When the IOU intersection-union ratio is greater than the preset threshold, the corresponding controls after the i-th control are assigned to the post-control group;

[0144] According to the number of controls included in the post-placed control group, the representative controls after duplicate removal are taken out, and the control set after duplicate removal includes a number of representative controls;

[0145] According to the number of controls in the post-placed control group, the representative controls after duplicate removal are taken out, and the control set after duplicate removal includes several representative controls, specifically:

[0146] When the rear control group contains more than two controls, the control corresponding to the second position in the rear control group is taken out as the representative control, otherwise the control corresponding to the first position in the rear control group is taken out as the representative control.

[0147] It should be noted that the controls in each control set are calculated and sorted according to their area. Then a relationship storage array is constructed to store the association relationship between controls. Duplicate controls are identified by calculating the IOU value, and duplicates are removed according to the preset threshold, and finally the representative control in each set is determined.

[0148] IOU (Intersection over Union) is an indicator that measures the degree of overlap between two spatial regions and is widely used in image processing, object detection, and machine vision. In the UI control association walkthrough method, IOU is used in the deduplication process to help identify and merge different elements representing the same control.

[0149] The calculation method of IOU intersection-and-union ratio is:

[0150] Given two spatial regions A and B, their intersection region is A∩B, and their union region is A∪B. The IOU intersection-union ratio can be calculated using the following formula:

[0151]

[0152] Among them, |A∩B| represents the area of ​​the overlapping part of regions A and B, and |A∪B| represents the total area covered by A and B.

[0153] When calculating the IOU of a UI control, the control is usually considered to be a rectangular area whose area can be calculated by the width and height of the control. If the control has an irregular shape, more complex geometric algorithms may be needed to determine the area of ​​the intersection and union.

[0154] By calculating the IOU value between controls, we can identify controls that are visually overlapping or very close to each other. These controls may be duplications caused by different HTML elements or style levels.

[0155] When the IOU value exceeds a certain preset threshold (e.g., 0.5 or higher), the two controls can be considered to be similar or different manifestations of the same entity, and thus they are merged in the deduplication process.

[0156] After deduplication, a representative control needs to be selected from the control group with a high IOU value. Usually, controls with a larger area or specific features are selected as representatives for use in subsequent association and comparison processes.

[0157] By removing duplicates, the number of controls that need to be processed in the subsequent association steps can be reduced, thereby improving the efficiency and accuracy of the association process.

[0158] An associating unit 204 is used to associate controls included in the control set after deduplication with each other according to preset priorities and coordinates to form a plurality of associated control sets corresponding to preset tags;

[0159] Further, the association unit 204 is specifically configured to:

[0160] Sort several representative controls contained in the control set in descending order according to the vertical coordinate;

[0161] Create a collection of associated controls;

[0162] The representative controls contained in the control collection are traversed in order, and the representative control combinations that are associated with each other are added to the associated control collection according to the horizontal coordinates and the preset priority differences.

[0163] Furthermore, the representative controls included in the control set are traversed in order, and according to the horizontal coordinates and the preset priority differences, the representative control combinations associated with each other are added to the associated control set. Specifically:

[0164] Traverse the representative controls contained in the control collection in order, and add the i-th representative control to the associated control collection;

[0165] Traverse the remaining representative controls after the ith representative control, and compare the horizontal coordinate differences between the target positions of the ith representative control and the remaining representative controls after the ith representative control and the preset priority differences on the front-end page in turn;

[0166] If there is a remaining representative control whose horizontal coordinate difference with the i-th representative control and the preset priority difference are both lower than the corresponding preset threshold, then the corresponding remaining representative control will be added to the associated control set as the associated control of the i-th representative control, otherwise after i=i+1, the representative controls contained in the traversal control set are returned.

[0167] Furthermore, the associating unit 204 is further configured to:

[0168] If the i-th representative component already exists in the associated control set, then i=i+1 and the representative controls contained in the control set are traversed again.

[0169] It should be noted that the representative controls after deduplication are sorted in descending order according to the ordinate (y coordinate). Then a set of associated controls is created, and two controls are associated according to the abscissa and the preset priority difference. During the association process, if the horizontal distance and priority difference of two controls are both lower than the preset threshold, they are considered to be associated and added to the associated control set.

[0170] If the horizontal coordinate difference and preset priority difference of two controls are both lower than the preset value, this usually means:

[0171] The two controls are close to each other in the horizontal direction. A small difference in the horizontal axis indicates that they are adjacent or closely arranged on the page, which may be an intentional arrangement based on design to keep the interface neat and logical;

[0172] Controls may be visually continuous or belong to the same group of functionally related elements. Smaller horizontal axis differences help maintain visual fluidity and consistency;

[0173] The preset priority is usually set according to the importance or functional ranking of the control in the page. If the difference between the priorities of two controls is small, it indicates that they may be related in function or belong to the same functional module.

[0174] The orderly arrangement and priority ranking of controls help maintain the consistency of the layout. When controls meet the conditions that the horizontal coordinate difference and priority difference are lower than the preset value, they may follow a certain design specification or layout pattern.

[0175] The comparison unit 205 is used to compare the associated control set according to the preset tags based on the preset control comparison rules to determine the problematic control.

[0176] It should be noted that the formed associated control set is compared according to the preset control comparison rules. These rules may involve the spacing and alignment between controls. After the comparison, it is determined which controls have problems and do not meet the design specifications.

[0177] Furthermore, it also includes:

[0178] The alarm unit is used to package the information of several problematic controls and send it to the target user for alarm.

[0179] The present application embodiment also provides another UI control association inspection device, such as Figure 3 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA), a sales terminal (English full name: Point of Sales, English abbreviation: POS), a car computer, etc., taking the terminal as a mobile phone as an example:

[0180] Figure 3 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal provided in an embodiment of the present application. Figure 3 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art can understand that Figure 3 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0181] Combine the following Figure 3 A detailed introduction to the various components of the mobile phone:

[0182] The RF circuit 1010 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1080 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (full name: Low Noise Amplifier, English abbreviation: LNA), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communications can use any communication standard or protocol, including but not limited to Global System of Mobile communications (Global System of Mobile communication, English abbreviation: GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, English abbreviation: CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (Long Term Evolution, English abbreviation: LTE), e-mail, Short Messaging Service (SMS), etc.

[0183] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0184] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1031 or near the touch panel 1031 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0185] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (full name in English: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (full name in English: Organic Light-Emitting Diode, English abbreviation: OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 3 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0186] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0187] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is output to the processor 1080 for processing, and then sent to another mobile phone through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing.

[0188] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1070. It provides users with wireless broadband Internet access. Figure 3 A WiFi module 1070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0189] The processor 1080 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 1020 and calling data stored in the memory 1020, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.

[0190] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0191] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0192] In the embodiment of the present application, the processor 1080 included in the terminal also has the following functions:

[0193] Step S1, capturing element information corresponding to all controls on the target page;

[0194] It should be noted that, for a target page that needs to be walked through, code can be injected into the target page to traverse the target page to capture element information corresponding to all controls.

[0195] Step S2: classify the element information according to preset labels to form a plurality of control sets corresponding to the preset labels;

[0196] It should be noted that the controls are classified according to preset tags in the element information corresponding to the captured controls to form several control sets.

[0197] If the control label is unclear or the control element is difficult to identify, you can use CSS selectors or other query methods to perform more precise queries based on other features of the control (such as position, style, content, etc.);

[0198] Computer vision technology can also be used to analyze page screenshots to identify control elements that are difficult to identify using traditional methods;

[0199] You can also train a machine learning model to identify and classify controls. The model can learn the characteristics of controls from the labeled data and gradually improve the recognition accuracy.

[0200] You can also build or optimize a rules engine that can identify and classify controls based on a complex set of rules and conditions.

[0201] Furthermore, controls that are difficult to identify are used as feedback to continuously iterate and optimize the recognition algorithms and rules.

[0202] Step S3, based on the area of ​​each control in the control set, performing deduplication processing on each control set respectively;

[0203] Specifically:

[0204] Step S31, calculating the area of ​​each control in a plurality of control sets;

[0205] Step S32, sorting the controls in the control set in descending order of area;

[0206] Step S33, constructing a relational storage array of a control set, wherein the relational storage array includes a front control and a rear control group;

[0207] Step S34, traverse in order, assign the i-th control to the preceding control;

[0208] Step S35, sequentially calculating the IOU intersection-over-union ratios between the area of ​​the ith control and the areas of the remaining controls after the ith control;

[0209] Step S36: When the IOU intersection-over-union ratio is greater than a preset threshold, the corresponding controls after the i-th control are assigned to the post-control group;

[0210] Step S37, according to the number of controls included in the post-control group, take out the representative controls after duplicate removal, and obtain a control set after duplicate removal including a number of representative controls;

[0211] Wherein, step S37 is specifically as follows:

[0212] When the rear control group contains more than two controls, the control corresponding to the second position in the rear control group is taken out as the representative control, otherwise the control corresponding to the first position in the rear control group is taken out as the representative control.

[0213] It should be noted that the controls in each control set are calculated and sorted according to their area. Then a relationship storage array is constructed to store the association relationship between controls. Duplicate controls are identified by calculating the IOU value, and duplicates are removed according to the preset threshold, and finally the representative control in each set is determined.

[0214] IOU (Intersection over Union) is an indicator that measures the degree of overlap between two spatial regions and is widely used in image processing, object detection, and machine vision. In the UI control association walkthrough method, IOU is used in the deduplication process to help identify and merge different elements representing the same control.

[0215] The calculation method of IOU intersection-and-union ratio is:

[0216] Given two spatial regions A and B, their intersection region is A∩B, and their union region is A∪B. The IOU intersection-union ratio can be calculated using the following formula:

[0217]

[0218] Among them, |A∩B| represents the area of ​​the overlapping part of regions A and B, and |A∪B| represents the total area covered by A and B.

[0219] When calculating the IOU of a UI control, the control is usually considered to be a rectangular area whose area can be calculated by the width and height of the control. If the control has an irregular shape, more complex geometric algorithms may be needed to determine the area of ​​the intersection and union.

[0220] By calculating the IOU value between controls, we can identify controls that are visually overlapping or very close to each other. These controls may be duplications caused by different HTML elements or style levels.

[0221] When the IOU value exceeds a certain preset threshold (e.g., 0.5 or higher), the two controls can be considered to be similar or different manifestations of the same entity, and thus they are merged in the deduplication process.

[0222] After deduplication, a representative control needs to be selected from the control group with a high IOU value. Usually, controls with a larger area or specific features are selected as representatives for use in subsequent association and comparison processes.

[0223] By removing duplicates, the number of controls that need to be processed in the subsequent association steps can be reduced, thereby improving the efficiency and accuracy of the association process.

[0224] Step S4: associate controls included in the control set after deduplication with each other according to preset priorities and coordinates to form a plurality of associated control sets corresponding to preset labels;

[0225] Specifically:

[0226] Step S41, sorting a number of representative controls included in the control set in descending order according to the vertical coordinates;

[0227] Step S42: Create an associated control set;

[0228] Step S43: traverse the representative controls included in the control set in order, and add the representative control combinations that are associated with each other to the associated control set according to the horizontal coordinates and the preset priority differences.

[0229] Further, step S43 is specifically as follows:

[0230] Step S431, traverse the representative controls included in the control set in order, and add the i-th representative control to the associated control set;

[0231] Step S432, traverse the remaining representative controls after the ith representative control, and compare the horizontal coordinate difference between the target positions of the ith representative control and the remaining representative controls after the ith representative control and the preset priority difference on the front page in turn;

[0232] Step S433: If there is a remaining representative control whose horizontal coordinate difference and preset priority difference with the i-th representative control are both lower than the corresponding preset threshold, then the corresponding remaining representative control will be added to the associated control set as the associated control of the i-th representative control; otherwise, set i=i+1 and return to execute step S431.

[0233] Furthermore, step S431 further includes:

[0234] If the i-th representative component already exists in the associated control set, then i=i+1 and step S431 is executed again.

[0235] It should be noted that the representative controls after deduplication are sorted in descending order according to the ordinate (y coordinate). Then a set of associated controls is created, and two controls are associated according to the abscissa and the preset priority difference. During the association process, if the horizontal distance and priority difference of two controls are both lower than the preset threshold, they are considered to be associated and added to the associated control set.

[0236] If the horizontal coordinate difference and preset priority difference of two controls are both lower than the preset value, this usually means:

[0237] The two controls are close to each other in the horizontal direction. A small difference in the horizontal axis indicates that they are adjacent or closely arranged on the page, which may be an intentional arrangement based on design to keep the interface neat and logical;

[0238] Controls may be visually continuous or belong to the same group of functionally related elements. Smaller horizontal axis differences help maintain visual fluidity and consistency;

[0239] The preset priority is usually set according to the importance or functional ranking of the control in the page. If the difference between the priorities of two controls is small, it indicates that they may be related in function or belong to the same functional module.

[0240] The orderly arrangement and priority ranking of controls help maintain the consistency of the layout. When controls meet the conditions that the horizontal coordinate difference and priority difference are lower than the preset value, they may follow a certain design specification or layout pattern.

[0241] Step S5: Based on the preset control comparison rules, the associated control set is compared according to the preset tags to determine the problematic control.

[0242] It should be noted that the formed associated control set is compared according to the preset control comparison rules. These rules may involve the spacing and alignment between controls. After the comparison, it is determined which controls have problems and do not meet the design specifications.

[0243] Furthermore, after step S5, the following steps are further included:

[0244] Step S6: Pack the information of several problematic controls and send it to the target user for warning.

[0245] It should be noted that the information of the problematic control is packaged and sent to the target user or development team for alerting. This is usually done by generating a report or directly highlighting the problematic control on the user interface.

[0246] An embodiment of the present application also provides a computer-readable storage medium for storing program code, wherein the program code is used to execute any one of the implementation methods of the UI control association walkthrough described in the aforementioned embodiments.

[0247] In the embodiments of the present application, a method, apparatus, device and storage medium for UI control association inspection are provided, which captures element information corresponding to all controls contained in a front-end page, classifies the controls according to preset labels, forms a control set, further deduplicates the control set based on the area of ​​each control in the control set, and finally associates the controls in pairs according to coordinates and preset priorities, determines problematic controls in the associated control set according to preset control comparison rules, implements conditional comparison between elements of multiple controls, provides more accurate spacing inspection, reduces manual inspection costs, and improves inspection efficiency.

[0248] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0249] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0250] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0251] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0252] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0253] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0254] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0255] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A UI control association walkthrough method, characterized in that: include: Step S1, capturing element information corresponding to all controls on the target page; Step S2: classify the element information according to preset labels to form a plurality of control sets corresponding to the preset labels; Step S3, based on the area of ​​each control in the control set, performing deduplication processing on each control set respectively; Step S4: associate the controls included in the control set after deduplication with each other according to preset priorities and coordinates to form a plurality of associated control sets corresponding to the preset labels; Step S5: Based on the preset control comparison rules, the associated control set is compared according to the preset tags to determine the problematic control.

2. The UI control association walkthrough method according to claim 1, characterized in that: The step S3 is specifically as follows: Step S31, calculating the area of ​​each control in a plurality of control sets; Step S32, sorting the controls in the control set in descending order of area; Step S33, constructing a relational storage array of the control set, wherein the relational storage array includes a front control and a rear control group; Step S34, assigning the ith control to the front control according to the sorting traversal; Step S35, sequentially calculating the IOU intersection-over-union ratios between the area of ​​the ith control and the areas of the remaining controls after the ith control; Step S36: when the IOU intersection-over-union ratio is greater than a preset threshold, assigning the corresponding control after the i-th control to the post-control group; Step S37, according to the number of controls included in the post-control group, take out the representative controls after duplicate removal, and obtain the control set after duplicate removal including a number of the representative controls.

3. The UI control association walkthrough method according to claim 2, characterized in that: The step S37 is specifically as follows: When the rear control group contains more than two controls, the control corresponding to the second position in the rear control group is taken out as the representative control, otherwise the control corresponding to the first position in the rear control group is taken out as the representative control.

4. The UI control association walkthrough method according to claim 2, characterized in that: The step S4 is specifically as follows: Step S41, sorting the plurality of representative controls included in the control set in descending order according to the vertical coordinates; Step S42: Create an associated control set; Step S43, traversing the representative controls included in the control set in order, and adding the representative control combinations associated with each other to the associated control set according to the horizontal coordinates and the preset priority differences.

5. The UI control association walkthrough method according to claim 4, characterized in that: The step S43 is specifically as follows: Step S431, traversing the representative controls included in the control set in order, and adding the i-th representative control to the associated control set; Step S432, traverse the remaining representative controls after the ith representative control, and compare the horizontal coordinate difference between the target position of the ith representative control and the remaining representative controls after the ith representative control and the preset priority difference on the front page in turn; Step S433: If there is a remaining representative control whose horizontal coordinate difference and the preset priority difference with the i-th representative control are both lower than the corresponding preset threshold, then the corresponding remaining representative control is added to the associated control set as an associated control of the i-th representative control; otherwise, set i=i+1 and return to execute step S431.

6. The UI control association walkthrough method according to claim 5, characterized in that: The step S431 further includes: If the i-th representative component already exists in the associated control set, then i=i+1 and the step S431 is executed again.

7. The UI control association walkthrough method according to claim 1, characterized in that: After step S5, the following steps are also included: Step S6: Pack the information of several problematic controls and send it to the target user for warning.

8. A UI control association walkthrough device, characterized in that: include: A capture unit is used to capture element information corresponding to all controls on the target page; A classification unit, used to classify the element information according to preset labels to form a plurality of control sets corresponding to the preset labels; A deduplication unit, used for performing deduplication processing on each control set based on the area of ​​each control in the control set; An associating unit, used to associate controls included in the control set after deduplication with each other according to preset priorities and coordinates, to form a plurality of associated control sets corresponding to the preset labels; The comparison unit is used to compare the associated control set according to the preset tags based on the preset control comparison rules to determine the problematic control.

9. A UI control association walkthrough device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the UI control association inspection method according to any one of claims 1 to 7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes are used to execute the UI control association walkthrough method according to any one of claims 1 to 7.

Images