Small program content detection method and system based on coordinate dot matrix

Abstract

The invention discloses an applet content detection method and system based on a coordinate dot matrix, and the method comprises the steps: carrying out the region division of an applet page based on the coordinate dot matrix according to a preset region specification, and obtaining a target region set of the applet page; according to the position of the target area, initializing the area object attribute of the target area; determining a target area from the target area set as a click object through a weighted random algorithm; carrying out click operation on the click object, updating the regional object attribute of the click object according to the change condition of the applet page, and carrying out screenshot processing on the changed page; and returning to the applet page before change, and re-executing the step of determining one target region from the target region set as the click object through the weighted random algorithm until a preset termination condition is met, thereby obtaining applet content. The method can realize accurate, rapid and high-universality applet content detection, and can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and system for detecting applet content based on a coordinate dot matrix. Background Art

[0002] In the related technology, when developing a mini-program, it is necessary to perform dial testing on the developed mini-program. During the dial testing, it is usually necessary to analyze the position of the clickable elements of the target mini-program, and then click the coordinate position through TouchAction and extract the content by taking a screenshot. This customized development method can only be applied to the dial testing tasks of a small number of mini-programs and is not very versatile. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a method and system for detecting mini-program content based on a coordinate dot matrix, aiming to achieve accurate, fast and highly versatile mini-program content detection.

[0004] To achieve the above purpose, one aspect of an embodiment of the present application proposes a method for detecting applet content based on a coordinate matrix, the method comprising:

[0005] According to the preset area specifications, the mini program page is divided into areas based on the coordinate dot matrix to obtain a target area set of the mini program page; wherein the target area set includes a plurality of target areas;

[0006] Initializing the region object attributes of the target region according to the location of the target region;

[0007] Determine a target area from the target area set as a click object by using a weighted random algorithm;

[0008] Perform a click operation on the click object, update the area object attribute of the click object according to the change of the mini program page, and take a screenshot of the page after the change;

[0009] Return to the mini-program page before the change, and re-execute the step of determining a target area from the target area set as a click object through a weighted random algorithm until a preset termination condition is reached to obtain the mini-program content.

[0010] In some embodiments, initializing the region object attribute of the target region according to the location of the target region comprises the following steps:

[0011] When the target area is located in the mini-program header area and / or the mobile device screen header area, marking the area attribute of the target area as a prohibited click area;

[0012] Calculating the region center coordinates of the target region according to the window specifications of the applet and the region specifications of the target region;

[0013] Initialize the jump page MD5 value set of the target area and the number of times the area is clicked.

[0014] In some embodiments, calculating the region center coordinates of the target region according to the window specifications of the applet and the region specifications of the target region comprises the following steps:

[0015] The horizontal coordinate is calculated based on the width of the mini program window, the total number of columns in the divided area, and the number of columns where the target area is located;

[0016] The vertical coordinate is calculated based on the mini program window height, the total number of rows in the divided area, and the number of rows where the target area is located;

[0017] The horizontal coordinate and the vertical coordinate are combined into the region center coordinate.

[0018] In some embodiments, the step of determining a target area from the target area set as a click object by using a weighted random algorithm comprises the following steps:

[0019] Obtaining the number of times each candidate area in the target area set is clicked; wherein the candidate area includes an undefined area and a clickable area;

[0020] Obtaining the total number of clicks on all the candidate areas in the target area set;

[0021] Calculate the weight value of each candidate region according to the number of times the region is clicked and the total number of times the region is clicked; wherein, the more times the region of the candidate region is clicked, the lower the weight value;

[0022] Randomly select one of the candidate areas with the highest weight value as the click object.

[0023] In some embodiments, the step of performing a click operation on the click object, updating the area object attribute of the click object according to the change of the mini-program page, and taking a screenshot of the changed page includes the following steps:

[0024] Get the first MD5 value of the mini program page before clicking;

[0025] Obtain the second MD5 value of the mini program page after clicking;

[0026] When the first MD5 value is consistent with the second MD5 value, it is determined that the mini program page has not changed before and after the click, and the area attribute in the area object attribute is updated to a static area;

[0027] When the first MD5 value is inconsistent with the second MD5 value, the screenshots before and after the click are compared for similarity using an image similarity algorithm to obtain image similarity;

[0028] When the image similarity is less than a preset similarity value, it is determined that the mini program page has changed before and after the click, and a screenshot of the page after the change is saved;

[0029] When there is a change before and after the click, the jump page MD5 value set of the area object attribute is obtained;

[0030] When the second MD5 value does not belong to the jump page MD5 value set, the jump page MD5 value set is updated according to the second MD5 value, the area attribute is updated to a clickable area, and the number of times the area is clicked in the area object attribute is automatically incremented.

[0031] In some embodiments, the preset termination condition includes at least one of the following:

[0032] After the click operation is performed on the click object, the mini program page does not change;

[0033] When the Mini Program page changes and the page has been visited after the change;

[0034] When the click operation is performed on the click object, the application software to which the mini-program belongs jumps out.

[0035] In some embodiments, the method further comprises the following steps:

[0036] According to the page MD5 value of the mini program page and all the target object attributes in the mini program page, a mapping relationship between the clicked positions of each target area is constructed;

[0037] According to the mapping relationship, a reachable relationship diagram between each of the mini-program pages is obtained;

[0038] Page jump path information is generated according to the reachable relationship graph and the region center coordinates of the target region.

[0039] To achieve the above purpose, another aspect of the embodiment of the present application provides a small program content detection system based on a coordinate lattice, the system comprising:

[0040] The first module is used to divide the mini program page into regions based on a coordinate matrix according to a preset region specification, and obtain a target region set of the mini program page; wherein the target region set includes a plurality of target regions;

[0041] The second module is used to initialize the area object attribute of the target area according to the location of the target area;

[0042] A third module is used to determine a target area from the target area set as a click object by using a weighted random algorithm;

[0043] The fourth module is used to click the click object, update the area object attribute of the click object according to the change of the mini program page, and take a screenshot of the page after the change;

[0044] The fifth module is used to return to the mini program page before the change, and re-execute the step of determining a target area from the target area set as a click object through a weighted random algorithm until a preset termination condition is reached to obtain the mini program content.

[0045] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0046] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0047] The embodiments of the present application include at least the following beneficial effects: The present application provides a method and system for detecting the content of a small program based on a coordinate dot matrix. The scheme divides the small program interface into a coordinate dot matrix and classifies it into clickable areas, static areas, and prohibited click areas. Click processing is performed on each coordinate dot matrix area based on recursion and weighted random algorithms, and the jumped page is judged using MD5 and image similarity algorithms. A screenshot of the new page is taken to obtain the content of the small program being tested. In general, by performing small program content detection using the method of the embodiments of the present application, the testing of a large number of different small programs can be completed quickly, saving development time. Through the weighted random algorithm, each coordinate point area can also be clicked, which is conducive to avoiding the omission of test elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0049] Figure 1 is a flow chart of a method for detecting applet content based on a coordinate lattice provided in an embodiment of the present application;

[0050] Figure 2 It is a step diagram of steps S210 to S230 provided in an embodiment of the present application;

[0051] Figure 3 It is a step diagram of steps S310 to S340 provided in an embodiment of the present application;

[0052] Figure 4 This is a flowchart of batch dialing test of a mini program provided in an embodiment of the present application;

[0053] Figure 5 is a flowchart of mini-program content detection provided by an embodiment of the present application;

[0054] Figure 6 This is a schematic diagram of segmenting a target area based on a coordinate lattice provided in an embodiment of the present application.

[0055] Figure 7 is a schematic diagram of jumping to a target area by clicking provided in an embodiment of the present application;

[0056] Figure 8 This is a schematic diagram of region attribute marking after all region content detections are completed provided in the embodiment of the present application;

[0057] Fig. 9 It is a module schematic diagram of a small program content detection system based on a coordinate lattice provided in an embodiment of the present application;

[0058] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0060] Although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100", "second / S200", etc. in the specification, claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0061] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0062] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0063] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0065] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0066] (1) Appium: Appium is a tool that inherits the automation concept of Selenium and aims to automate mobile app operations;

[0067] (2) Content Dialing: refers to the method of obtaining resource information such as system interface text, images, and videos in the field of information security.

[0068] (3) Mini Program: This refers to the mini program opened in the application software (APP);

[0069] (4)TouchAction: Appium mobile terminal automated testing of touch controls.

[0070] In the related technology, when developing a mini-program, it is necessary to perform dial testing on the developed mini-program. During the dial testing, it is usually necessary to analyze the position of the clickable elements of the target mini-program, and then click the coordinate position through TouchAction and extract the content by taking a screenshot. This customized development method can only be applied to dial testing tasks for a small number of mini-programs. If there are hundreds or thousands of mini-program dial testing tasks, it is very time-consuming and troublesome to perform coordinate click analysis and screenshots for each different mini-program interface.

[0071] In view of this, a method and system for detecting the content of a small program based on a coordinate dot matrix is ​​provided in an embodiment of the present application. The scheme can be applied to the content detection of the small program in the application to realize the dialing test of the small program. First, the interface of the small program is divided into a coordinate dot matrix and classified into a clickable area, a static area and a prohibited click area. Among them, the clickable area refers to the area where the interface will change after clicking; the static area refers to the area that will not change after clicking; the prohibited click area may exit the small program after clicking. Then, click processing based on recursion and weighted random algorithms is performed on each coordinate dot matrix area, and MD5 and image similarity algorithms are used to judge the jumped page, and a screenshot of the new page is taken to obtain the content of the small program being dialed. In general, by performing the content detection of the small program through the method of the embodiment of the present application, the dialing test of a large number of different small programs can be completed quickly, saving development time. Through the weighted random algorithm, each coordinate point area can also be clicked, which is conducive to avoiding the omission of dialing elements.

[0072] A small program content detection method based on a coordinate lattice provided in an embodiment of the present application relates to the field of computer technology. A small program content detection method based on a coordinate lattice provided in an embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in a blockchain network; the software can be an application that implements a small program content detection method based on a coordinate lattice, etc., but is not limited to the above forms.

[0073] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0074] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0075] Figure 1 is an optional flow chart of a method for detecting applet content based on a coordinate lattice provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S100 to S500.

[0076] Step S100, according to preset area specifications, the mini program page is divided into areas based on a coordinate matrix to obtain a target area set of the mini program page; wherein the target area set includes a plurality of target areas.

[0077] Step S200: initializing the region object attributes of the target region according to the location of the target region.

[0078] Step S300: determining a target area from the target area set as a click object by using a weighted random algorithm.

[0079] Step S400, perform a click operation on the click object, update the area object properties of the click object according to the changes of the mini-program page, and take a screenshot of the changed page.

[0080] Step S500, return to the mini program page before the change, and re-execute the step of determining a target area from the target area set as a click object through a weighted random algorithm until a preset termination condition is reached to obtain the mini program content.

[0081] Steps S100 to S500 shown in the embodiment of the present application divide the mini-program interface into a coordinate dot matrix and classify them into clickable areas, static areas, and prohibited click areas. Click processing is performed on each coordinate dot matrix area based on recursion and weighted random algorithms, and the jumped page is judged using MD5 and image similarity algorithms, and a screenshot of the new page is taken to obtain the content of the mini-program being tested. In general, by performing mini-program content detection using the method of the embodiment of the present application, it is possible to quickly complete the testing of a large number of different mini-programs, saving development time. Through the weighted random algorithm, it is also possible to make each coordinate point area likely to be clicked, which is beneficial to avoid missing test elements.

[0082] In step S100 of some embodiments, the preset area specifications can be adjusted according to the layout of the mini-program. When dividing the area, it is also necessary to ensure that a click on an area will trigger at most one event. The smaller the area is divided, the more accurate the mini-program content obtained.

[0083] See also Figure 2 In some embodiments, step S200 includes but is not limited to the following steps S210 to S230:

[0084] Step S210: When the target area is located in the mini-program header area and / or the mobile device screen header area, the area attribute of the target area is marked as a prohibited click area.

[0085] Step S220, calculating the region center coordinates of the target region according to the window specifications of the applet and the region specifications of the target region.

[0086] Step S230, initializing the jump page MD5 value set of the target area and the number of times the area is clicked.

[0087] Initializing the forbidden click area, the area center coordinates, the jump page MD5 value set and the number of times the area is clicked through steps S210 to S230 is beneficial to the subsequent target area selection and page jump judgment.

[0088] In some embodiments, step S220 includes but is not limited to the following steps S221 to S223:

[0089] Step S221, calculating the horizontal coordinate according to the width of the mini-program window, the total number of columns of the divided areas, and the number of columns where the target area is located.

[0090] Step S222, calculating the vertical coordinate according to the mini-program window height, the total number of rows of the divided areas, and the number of rows where the target area is located.

[0091] Step S223, combining the horizontal coordinate and the vertical coordinate into the region center coordinate.

[0092] By calculating the center coordinates of the region in steps S221 to S223 and using the center coordinates of the region based on the coordinate lattice to represent the target region, the amount of calculation can be effectively reduced, computing resources can be saved, and computing efficiency can be improved.

[0093] See also Figure 3 In some embodiments, the step S300 includes but is not limited to the following steps S310 to S340:

[0094] Step S310, obtaining the number of times each candidate area in the target area set is clicked; wherein the candidate area includes an undefined area and a clickable area.

[0095] Step S320: Obtain the total number of clicks on all the candidate areas in the target area set.

[0096] Step S330, calculating a weight value of each candidate region according to the number of times the region is clicked and the total number of times the region is clicked; wherein, the more times the region of the candidate region is clicked, the lower the weight value.

[0097] Step S340: randomly select one of the candidate areas with the highest weight value as the click object.

[0098] Through steps S310 to S340, a weighted random algorithm is used to select a candidate area as a click object, and the selected area can be controlled so that the area with a higher number of clicks is less likely to be selected, which is conducive to clicking all areas of the page within a certain period of time.

[0099] In some embodiments, step S400 includes but is not limited to the following steps S410 to S470:

[0100] Step S410, obtaining the first MD5 value of the mini program page before clicking.

[0101] Step S420, obtaining the second MD5 value of the mini program page after clicking.

[0102] Step S430: When the first MD5 value is consistent with the second MD5 value, it is determined that the mini-program page has no change before and after the click, and the area attribute in the area object attribute is updated to a static area.

[0103] Step S440: When the first MD5 value is inconsistent with the second MD5 value, the screenshots before and after the click are compared for similarity using an image similarity algorithm to obtain image similarity.

[0104] Step S450, when the image similarity is less than a preset similarity value, it is determined that the mini program page has changed before and after being clicked, and a screenshot of the page after the change is saved.

[0105] Step S460: when there is a change before and after the click, a jump page MD5 value set of the area object attribute is obtained.

[0106] Step S470, when the second MD5 value does not belong to the jump page MD5 value set, update the jump page MD5 value set according to the second MD5 value, update the area attribute to a clickable area, and increment the number of times the area is clicked in the area object attribute.

[0107] By comparing the MD5 values ​​of the page before and after clicking through the above steps S410 to S470, it is possible to quickly confirm whether the page has changed. Furthermore, by further using an image similarity algorithm to compare the similarity of the screenshots before and after the jump, it is possible to prevent situations such as the time displayed at the top of the mobile phone page, characters entered in the input box, etc., which may cause minor changes in the page and lead to inconsistent MD5 values, thereby making the judgment of page changes more accurate.

[0108] In step S500 of some embodiments, the preset termination condition of step S500 includes at least one of the following conditions a to c:

[0109] a. After the click operation is performed on the click object, the mini program page does not change.

[0110] b. When the mini program page changes and the page has been visited after the change.

[0111] c. After the click operation is performed on the click object, the application software to which the mini-program belongs pops up.

[0112] The above preset termination conditions can prevent page jumps from falling into an infinite loop.

[0113] In some embodiments, the method may further include the following steps S610-S630:

[0114] Step S610: construct a mapping relationship between clicked positions of each target area according to the page MD5 value of the mini program page and all target object attributes in the mini program page.

[0115] Step S620: obtaining a reachable relationship diagram between the mini-program pages according to the mapping relationship.

[0116] Step S630: Generate page jump path information according to the reachable relationship graph and the region center coordinates of the target region.

[0117] Generating path information through steps S610 to S630 is helpful for page tracing.

[0118] Below, in conjunction with a specific application example of a mini-program content dialing scenario, the scheme of the embodiments of the present application is introduced and explained in detail. It should be noted that the actual numerical values ​​in the following embodiments are only used as an example, and this application is not limited to this.

[0119] In an embodiment of the present application, a method for detecting the content of a mini-program based on a coordinate matrix is ​​provided, which can be applied to detect the content of a mini-program in a mini-program dialing task. Furthermore, the TouchAction component can be used to click on the coordinates of the mini-program interface and capture a screenshot of the interface after the change to perform content dialing, and reduce the number of clicks on repeated target locations when obtaining the coordinates of interface elements. Specifically, refer to Figure 4 When a large number of mini-programs need to be dialed, obtain the dialing task list from the database, obtain the mini-program name from the task list, connect to the appium service, link the mobile device through appium, and search for the application software, enter the mini-program management interface of the application software, search for the mini-program according to the task name, and after finding the mini-program, execute the mini-program content detection method based on coordinate lattice proposed in the embodiment of the present application, save the obtained mini-program screenshot content to the database, and push it to the relevant mini-program dialing review platform for machine review and manual review. When the specified dialing time is reached, the dialing task is completed.

[0120] Further, refer to Figure 5 The method for detecting the content of a small program based on a coordinate matrix proposed in the embodiment of the present application can be implemented as follows (1) to (5):

[0121] 1. According to the preset area specifications, the mini program page is divided into areas based on the coordinate dot matrix to obtain a target area set of the mini program page; wherein the target area set includes a plurality of target areas.

[0122] like Figure 6 As shown, Figure 6Areas 1-6 in the chart are configured as prohibited click areas because these areas are the head areas of the mini-program and mobile device (such as mobile phone) screens. The interface elements in this area are generally unrelated to the mini-program content. For example, Figure 6 Area 6 in the dialog box will exit the applet.

[0123] Will Figure 6 Area 7-54 in the diagram is configured as a clickable area and a static area. The two cannot be distinguished here. The area that cannot be distinguished temporarily can be called an undefined area. It is necessary to mark it according to the page changes during subsequent dialing.

[0124] The specifications of these areas can be changed according to the layout specifications of the target applet interface. In theory, the smaller the area is divided, the more likely each click area will be clicked. However, if the area is divided too small, the amount of calculation for repeated clicks will increase. The area specifications can be configured according to the layout of the target applet interface to balance. In addition, the divided areas must also meet the requirement that only one type of click event will be triggered after the motor is turned on in one area.

[0125] 2. Initialize the region object properties of the target region according to the location of the target region.

[0126] After creating the area, configure the area object properties for each target area. The area object properties include the coordinates of the center point of the area, the jump page MD5 value set, the number of times the area is clicked, and the area properties. Among them, the area properties are divided into forbidden click areas, clickable areas, static areas, and undefined areas; the jump page MD5 value set is the MD5 value of the page that changes when the click page changes. There may be multiple MD5 values, so it can be represented by an array.

[0127] Exemplarily, the data structure of the region object attribute can be designed as follows:

[0128] Area number: {Point_Y,Point_X,NextPageMD5List:[MD51,MD52],TouchTimes,Type};

[0129] Among them, Point_Y represents the Y coordinate of the center point of the area; Point_X represents the X coordinate of the center point of the area; NextPageMD5List:[MD51,MD52] represents the MD5 value array of the jump interface. If the interface changes when the area is clicked, the MD5 value record of the changed interface is recorded, and there may be multiple records (such as the above MD51 and MD52); TouchTimes represents the number of times the area is clicked; Type represents the area attribute, which is divided into prohibited click areas (yellow mark: Yellow), clickable areas (red mark: Red), static areas (green mark: Green), and undefined areas (markers are set to empty or other colors).

[0130] In some embodiments, the calculation formula for the X coordinate is:

[0131]

[0132] The calculation formula for the Y coordinate is:

[0133]

[0134] Among them, windowWide represents the width of the mini program window; sumColumn represents the total number of columns in the divided area; nowColumn represents the column number where the target area is located; windowHeight represents the height of the mini program window; sumLine represents the total number of rows in the divided area; nowLine represents the row number where the target area is located.

[0135] Furthermore, the mapping relationship between the mini program page and all target areas of the page can be configured. Exemplarily, the configuration can be as follows:

[0136] Mini Program homepage MD5 value: [

[0137] Area 1:{Point_Y:y1,Point_X:x1,NextPageMD5List:[],TouchTimes:0,Type:Yellow},

[0138] Area 2:{Point_Y:y2,Point_X:x2,NextPageMD5List:[],TouchTimes:0,Type:Yellow},

[0139] …

[0140] Area 54: {Point_Y:y54,Point_X:x54,NextPageMD5List:[],TouchTimes:0,Type:NULL}].

[0141] 3. Determine a target area as the click object from the target area set through a weighted random algorithm.

[0142] After the above steps 1 and 2, the initialization phase of the applet dial test has been completed. Next, the scanning task of the target area is carried out. Specifically, refer to Figure 7 The page jump of the mini program homepage and the scanning task can be implemented as follows: 3.1~3.2:

[0143] 3.1. If the page does not change after clicking area 14, set the Type value of area 14 corresponding to the MD5 of the mini program homepage to Green and mark area 14 under the page as unclickable. Then return to the previous page of the mini program. If it is the mini program homepage, there is no need to return.

[0144] After step 3.1, the region object attributes are as follows:

[0145] Area 14: {Point_Y:y1,Point_X:x1,NextPageMD5List:[],TouchTimes:0,Type:Green}.

[0146] 3.2. If the page changes after clicking on area 14, set the Type value of area 14 corresponding to the MD5 of the mini program homepage to Red to mark area 14 under the page as clickable, set the TouchTimes field of the area to add 1, and store the MD5 value of the page after the jump: xxxx2 in the NextPageMD5List collection, and take a screenshot of the interface after the jump through the Appium control, and push the screenshot content to the audit platform. The TouchTimes field is mainly used for weighted randomness. The higher the number of TouchTimes fields, the lower the weight, and the lower the probability of the next area being selected. This ensures that all areas of the page can be clicked within a certain period of time.

[0147] After step 3.2, the properties of the region object are as follows:

[0148] Area 14: {Point_Y:y14,Point_X:x14,NextPageMD5List:['xxxx2'],TouchTimes:1,Type:Red}.

[0149] 4. Click the click object, update the area object properties of the click object according to the changes of the mini program page, and take a screenshot of the page after the change.

[0150] In the embodiment of the present application, the above steps are repeated on the xxxx2 page according to the recursive idea to establish the target click area data structure, and select the click according to the random algorithm. The recursive termination conditions of the area based on the dot matrix coordinates include the following ①~③:

[0151] ① When the interface does not change after clicking the area (compare the MD5 values ​​of the interface before and after clicking and confirm the result with the image similarity algorithm), return to the previous page (no need to return to the home page).

[0152] ② When the interface changes after clicking the area, but the changed MD5 value is already in the area NextPageMD5List list, it returns to the previous page.

[0153] ③After clicking the area, determine whether the current target application is still in the application to prevent it from crossing the boundary and escaping the application applet. If not, reopen the applet homepage for dialing test.

[0154] 5. After running the above steps for a period of time, a data structure of the mapping relationship can be obtained. The jump page MD5 value field of this structure records the reachable relationship diagram between the various pages of the mini program and the jump path information generated according to the regional coordinates of each interface, providing strong support for page tracing.

[0155] For example, Figure 8 The data structure of the mapping relationship after running for a period of time is as follows:

[0156] Mini Program Home Page MD5_1: [

[0157] Area 1:{Point_Y:y1,Point_X:x1,NextPageMD5List:[],TouchTimes:0,Type:Yellow},

[0158] Area 2:{Point_Y:y2,Point_X:x2,NextPageMD5List:[],TouchTimes:0,Type:Yellow},

[0159] Area 3:{Point_Y:y3,Point_X:x3,NextPageMD5List:[],TouchTimes:0,Type:Yellow},

[0160] Area 4:{Point_Y:y4,Point_X:x4,NextPageMD5List:[],TouchTimes:0,Type:Yellow},

[0161] Area 5:{Point_Y:y5,Point_X:x5,NextPageMD5List:[],TouchTimes:0,Type:Yellow},

[0162] Area 6:{Point_Y:y6,Point_X:x6,NextPageMD5List:[],TouchTimes:0,Type:Yellow},

[0163] Area 7:{Point_Y:y7,Point_X:x7,NextPageMD5List:['jump to page MD51','jump to page MD52'],TouchTimes:2,Type:Red},

[0164] Area 8:{Point_Y:y8,Point_X:x8,NextPageMD5List:['Jump to page MD51','Jump to page MD52','Jump to page MD53'],TouchTimes:3,Type:Red},

[0165] Area 9:{Point_Y:y9,Point_X:x9,NextPageMD5List:['Jump to page MD51','Jump to page MD52'.],TouchTimes:2,Type:Red},

[0166] Area 10:{Point_Y:y10,Point_X:x10,NextPageMD5List:[],TouchTimes:1,Type:Green},

[0167] …

[0168] Area 54: {Point_Y: y54, Point_X: x54, NextPageMD5List: ['Jump to page MD52'], TouchTimes: 1, Type: Red}]

[0169] MD5_2 of subpage 1: [...],

[0170] MD5_3 of subpage 2: [...].

[0171] Among them, the data structure of the sub-page is similar to the above-mentioned homepage and will not be repeated here.

[0172] In some embodiments, taking the data of the mini program homepage as an example, the steps for implementing the weighted random algorithm include:

[0173] (1) Take the areas with Type type NULL (i.e., undefined area) and RED (clickable area) as candidate areas. The obtained candidate area list is: [area 7, area 8, area 9, area 10, area 11, area 12, area 13, area 14, area 15, area 54].

[0174] (2) Get the TouchTimes field of all candidate areas: [6, 0, 1, 0, 0, 0, 0, 2, 1, 0].

[0175] (3) Define the weight of each candidate region:

[0176] First, calculate the total number of clicks for all candidate areas, and subtract the number of clicks for each candidate area from the total number of clicks to get the difference for each candidate area. Calculate the sum of all differences, and then calculate the weight of each area based on the sum of differences. For example: if the total number of clicks SumTemp_TouchTimes = 6+0+1+0+0+0+0+2+1+0 = 10; subtract the TouchTimes in each candidate area from SumTemp_TouchTimes to get: [4, 10, 9, 10, 10, 10, 8, 9, 10]; calculate the current total number of Sum_TouchTimes: Sum_TouchTimes = 4+10+9+10+10+10+10+8+9+10 = 90; and then calculate the weight of each area:

[0177] [4 / 90,10 / 90,9 / 90,10 / 90,10 / 90,10 / 90,10 / 90,8 / 90,9 / 10 / ,10 / 10]=[0.04,0.11,0.1,0.11,0.11,0.11,0.11,0.08,0.1,0.11].

[0178] (4) According to the weight calculation result, a region is randomly selected from the candidate regions with the highest weight as the click object. This algorithm can ensure that the region with the highest weight can be obtained every time. In other words, the region that has not been clicked is selected first.

[0179] In some embodiments, since the time is often displayed above the mobile phone page, the MD5 of each screenshot page will change. Here, the fixed position and size of the screenshot can be set to bypass these uncertain information and obtain accurate image MD5 information.

[0180] Some click buttons may be input boxes. Each click will input characters, causing slight changes to the page, which may cause the page to loop. The image similarity comparison algorithm can be used to solve these problems.

[0181] Specifically, you can first determine whether the MD5 values ​​before and after the click are consistent. If they are consistent, the click did not trigger a page change; if the MD5 values ​​are inconsistent, use an image similarity comparison algorithm to compare the screenshots before and after the click to obtain the image similarity. When the image similarity is less than the preset similarity value, it is determined that there is a change in the mini program page before and after the click.

[0182] In summary, the embodiments of the present application at least include the following beneficial effects:

[0183] 1. It can provide a general solution for quick dialing content for large-scale dialing of different mini-programs, so that when developing mini-programs, there is no need to develop a separate script program for each mini-program, saving development time.

[0184] 2. The weighted random algorithm ensures that every area can be clicked, avoiding the omission of dialing elements.

[0185] 3. By establishing a data structure that associates the page with the jump page, it is easier to trace the interface path.

[0186] 4. Prevent duplicate mini-program screenshots from being pushed to the review platform by comparing the jump page MD5 value to see if it is in the jump page MD5 value set.

[0187] See also Fig. 9 The embodiment of the present application further provides a small program content detection system based on a coordinate lattice, which can implement the above-mentioned small program content detection method based on a coordinate lattice, and the system includes:

[0188] The first module 101 is used to divide the mini program page into regions based on a coordinate matrix according to a preset region specification, and obtain a target region set of the mini program page; wherein the target region set includes a plurality of target regions;

[0189] The second module 102 is used to initialize the area object attribute of the target area according to the location of the target area;

[0190] The third module 103 is used to determine a target area from the target area set as a click object by using a weighted random algorithm;

[0191] The fourth module 104 is used to perform a click operation on the click object, update the area object attribute of the click object according to the change of the mini program page, and take a screenshot of the changed page;

[0192] The fifth module 105 is used to return to the mini-program page before the change, and re-execute the step of determining a target area from the target area set as a click object through a weighted random algorithm until a preset termination condition is reached to obtain the mini-program content.

[0193] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0194] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned method for detecting the content of a small program based on a coordinate dot matrix when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0195] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0196] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0197] The processor 201 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0198] The memory 202 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 202 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 202, and the processor 201 calls and executes a small program content detection method based on a coordinate lattice in the embodiment of this application;

[0199] Input / output interface 203, used to implement information input and output;

[0200] The communication interface 204 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0201] Bus 205 , which transmits information between various components of the device (e.g., processor 201 , memory 202 , input / output interface 203 , and communication interface 204 );

[0202] The processor 201 , the memory 202 , the input / output interface 203 and the communication interface 204 are connected to each other in communication within the device via the bus 205 .

[0203] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned method for detecting applet content based on a coordinate dot matrix.

[0204] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0205] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0206] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0207] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0208] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0209] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0210] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0211] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0212] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0213] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0214] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0215] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0216] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for detecting the content of a small program based on a coordinate matrix, characterized in that: The following steps are involved: According to the preset area specifications, the mini program page is divided into areas based on the coordinate dot matrix to obtain a target area set of the mini program page; wherein the target area set includes a plurality of target areas; Initializing the region object attributes of the target region according to the location of the target region; Determine a target area from the target area set as a click object by using a weighted random algorithm; Perform a click operation on the click object, update the area object attribute of the click object according to the change of the mini program page, and take a screenshot of the page after the change; Return to the mini-program page before the change, and re-execute the step of determining a target area from the target area set as a click object through a weighted random algorithm until a preset termination condition is reached to obtain the mini-program content.

2. The method according to claim 1, characterized in that Initializing the area object attributes of the target area according to the location of the target area includes the following steps: When the target area is located in the mini-program header area and / or the mobile device screen header area, marking the area attribute of the target area as a prohibited click area; Calculating the region center coordinates of the target region according to the window specifications of the applet and the region specifications of the target region; Initialize the jump page MD5 value set of the target area and the number of times the area is clicked.

3. The method according to claim 1, characterized in that The step of calculating the region center coordinates of the target region according to the window specifications of the applet and the region specifications of the target region comprises the following steps: The horizontal coordinate is calculated based on the width of the mini program window, the total number of columns in the divided area, and the number of columns where the target area is located; The vertical coordinate is calculated based on the mini program window height, the total number of rows in the divided area, and the number of rows where the target area is located; The horizontal coordinate and the vertical coordinate are combined into the region center coordinate.

4. The method according to claim 1, characterized in that: The step of determining a target area from the target area set as a click object by using a weighted random algorithm comprises the following steps: Obtaining the number of times each candidate area in the target area set is clicked; wherein the candidate area includes an undefined area and a clickable area; Obtaining the total number of clicks on all the candidate areas in the target area set; Calculate the weight value of each candidate region according to the number of times the region is clicked and the total number of times the region is clicked; wherein, the more times the region of the candidate region is clicked, the lower the weight value; Randomly select one of the candidate areas with the highest weight value as the click object.

5. The method according to claim 1, characterized in that The step of performing a click operation on the click object, updating the area object attribute of the click object according to the change of the mini-program page, and taking a screenshot of the changed page includes the following steps: Get the first MD5 value of the mini program page before clicking; Obtain the second MD5 value of the mini program page after clicking; When the first MD5 value is consistent with the second MD5 value, it is determined that the mini program page has not changed before and after the click, and the area attribute in the area object attribute is updated to a static area; When the first MD5 value is inconsistent with the second MD5 value, the screenshots before and after the click are compared for similarity using an image similarity algorithm to obtain image similarity; When the image similarity is less than a preset similarity value, it is determined that the mini program page has changed before and after the click, and a screenshot of the page after the change is saved; When there is a change before and after the click, the jump page MD5 value set of the area object attribute is obtained; When the second MD5 value does not belong to the jump page MD5 value set, the jump page MD5 value set is updated according to the second MD5 value, the area attribute is updated to a clickable area, and the number of times the area is clicked in the area object attribute is automatically incremented.

6. The method according to claim 1, characterized in that The preset termination condition includes at least one of the following: After the click operation is performed on the click object, the mini program page does not change; When the Mini Program page changes and the page has been visited after the change; When the click operation is performed on the click object, the application software to which the mini-program belongs jumps out.

7. The method according to claim 1, characterized in that The method further comprises the following steps: According to the MD5 value of the mini program page and all the target object attributes in the mini program page, a mapping relationship between the clicked positions of each target area is constructed; According to the mapping relationship, a reachable relationship diagram between each of the mini-program pages is obtained; Page jump path information is generated according to the reachable relationship graph and the region center coordinates of the target region.

8. A small program content detection system based on coordinate dot matrix, characterized in that: include: The first module is used to divide the mini program page into regions based on a coordinate matrix according to a preset region specification, and obtain a target region set of the mini program page; wherein the target region set includes a plurality of target regions; The second module is used to initialize the area object attribute of the target area according to the location of the target area; A third module is used to determine a target area from the target area set as a click object by using a weighted random algorithm; The fourth module is used to click the click object, update the area object attribute of the click object according to the change of the mini program page, and take a screenshot of the page after the change; The fifth module is used to return to the mini program page before the change, and re-execute the step of determining a target area from the target area set as a click object through a weighted random algorithm until a preset termination condition is reached to obtain the mini program content.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.

Images