Page change recognition method and apparatus, electronic apparatus, and storage medium

Abstract

The present application relates to a page change recognition method and apparatus, an electronic apparatus, and a storage medium. The page change recognition method comprises: acquiring page training data of a mobile terminal device, and inputting the page training data separately into a preset first model to be trained and a preset second model to be trained for training to obtain a model under test, wherein initialization parameters of said first model are different from those of said second model; inputting page test data of the mobile terminal device to the model under test, so as to test the model under test to obtain a target recognition model, wherein said first model, said second model and the model under test each comprise a target page attention module, and the number of output channels of a convolution kernel in the target page attention module is determined by the number of input channels of the convolution kernel; and inputting a page to be recognized into the target recognition model so as to recognize whether a change occurs in the page. The present application facilitates improvement of the accuracy of page change recognition.

Description

A method, apparatus, electronic device, and storage medium for identifying page changes. Technical Field

[0001] This application relates to the field of data processing, and in particular to a method, apparatus, electronic device, and storage medium for recognizing page changes. Background Technology

[0002] The APP content testing project involves testing hundreds of relevant apps to detect any inappropriate content on their pages. The general testing process involves automating operations on the app, acquiring the text content and screenshots of the page, and finally using a review model to review the text content and screenshots. Among the automated operations, determining whether the page has changed before and after a certain action is crucial. If this cannot be effectively determined, the automated operation may get stuck on a specific page and repeatedly acquire information from the same page. This is fatal in APP content testing, as it prevents the detection of new pages and places a significant burden on subsequent content review processes.

[0003] Currently, determining whether an app page has changed primarily involves comparing the page source code before and after an Appium operation to see if they are consistent, and comparing the hash similarity of the page images before and after the Appium operation. However, this method of determining page change by comparing page source code and hash similarity can lead to false positives, resulting in poor accuracy in page change detection.

[0004] There is currently no effective solution to the problem of poor accuracy in page change recognition in related technologies. Summary of the Invention

[0005] This embodiment provides a method, apparatus, electronic device, and storage medium for identifying page changes, in order to solve the problem of poor accuracy in page change identification in related technologies.

[0006] Firstly, this embodiment provides a method for recognizing page changes, applied to a terminal device, the method comprising:

[0007] The page training data of the mobile device is obtained, and the page training data is input into the preset first training model and the second training model respectively for training to obtain the test model; the initialization parameters of the first training model and the second training model are different.

[0008] The page test data of the mobile device is input into the model to be tested to test the model and obtain the target recognition model; the first model to be trained, the second model to be trained, and the model to be tested all include a target page attention module; the number of output channels of the convolution kernel in the target page attention module is determined by the number of input channels of the convolution kernel;

[0009] Input the page to be identified into the target recognition model to identify whether the page has changed.

[0010] In some embodiments, the step of acquiring page training data from a mobile device and inputting the page training data into a preset first training model and a preset training model for training includes:

[0011] Retrieve attribute data from the page's training data;

[0012] A preset page change rate is used to determine the page change area based on the page change rate and the attribute data.

[0013] The page change area is filled to obtain the first target page training data, and the first target page training data is input into the first model to be trained.

[0014] In some embodiments, the step of acquiring page training data from a mobile device and inputting the page training data into a preset first training model and a preset training model for training includes:

[0015] The training data of the page is deep copied to obtain the training data of the second target page;

[0016] Input the training data of the second target page into the second model to be trained.

[0017] In some embodiments, the step of acquiring page training data from a mobile device and inputting the page training data into a preset first training model and a second training model for training to obtain a test model includes:

[0018] Input the first target page training data into the first model to be trained to obtain the first training output data, and input the second target page training data into the second model to be trained to obtain the second training output data;

[0019] Determine the cosine distance between the first training output data and the second training output data;

[0020] The model loss is determined based on the cosine distance and the page change rate. The first and second training models are then updated based on the model loss to obtain the test model.

[0021] In some embodiments, inputting page test data from the mobile device into the model under test to test the model and obtain a target recognition model includes:

[0022] The page test data is preprocessed according to the preset page change rate to obtain the first page test data and the second page test data.

[0023] Input the first page test data and the second page test data into the model to be tested, respectively, to obtain the first page test results and the second page test results;

[0024] Determine the cosine distance between the test results of the first page and the test results of the second page, and obtain the test similarity based on the cosine distance;

[0025] The test accuracy of the model under test is determined based on whether the difference between the page change rate and the test similarity exceeds a preset tolerance threshold.

[0026] The target recognition model is determined based on the test accuracy.

[0027] In some embodiments, the step of inputting the page to be identified to the target recognition model to identify whether the page has changed includes:

[0028] Obtain the features of the page to be identified, and determine whether the features are consistent with the preset initialization parameter features;

[0029] If they match, it is determined that the page to be identified has changed; if they do not match, the cosine distance between the feature and the initialization parameter feature is determined, and the page is determined to have changed based on the cosine distance.

[0030] In some embodiments, the step of inputting the page to be identified into the target recognition model to identify whether the page has changed further includes:

[0031] The preset initialization parameter features are updated based on the features of the page to be identified.

[0032] Secondly, this embodiment provides a page change recognition device, which includes a training module, a testing module, and a recognition module;

[0033] The training module is used to acquire page training data from the mobile device, and input the page training data into a preset first training model and a second training model for training to obtain a test model; the initialization parameters of the first training model and the second training model are different.

[0034] The testing module is used to input page test data from the mobile device into the model to be tested to test the model and obtain a target recognition model; the first model to be trained, the second model to be trained, and the model to be trained all include a target page attention module; the number of output channels of the convolution kernel in the target page attention module is determined by the number of input channels of the convolution kernel;

[0035] The recognition module is used to input the page to be recognized into the target recognition model in order to identify whether the page has changed.

[0036] Thirdly, this embodiment provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the page change recognition method described in the first aspect above.

[0037] Fourthly, this embodiment provides a storage medium storing a computer program that, when executed by a processor, implements the page change recognition method described in the first aspect above.

[0038] Compared with related technologies, the page change recognition method, apparatus, electronic device and storage medium provided in this embodiment trains page training data using a parallel dual-model training model including a target page attention module to obtain a test model. Then, based on the test model, page test data is tested to obtain a target recognition model. Finally, page change recognition is performed based on a serial single-target recognition model, which helps to improve the accuracy of page change recognition.

[0039] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0040] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0041] Figure 1 is a hardware structure block diagram of the terminal of the page change recognition method according to an embodiment of this application;

[0042] Figure 2 is a flowchart of the page change recognition method provided in an embodiment of this application;

[0043] Figure 3 is a flowchart of a page change recognition method based on deep learning provided in an embodiment of this application;

[0044] Figure 4 is a schematic diagram of a hardware structure provided in this specific embodiment;

[0045] Figure 5 is a structural diagram of a page attention structure provided in this specific embodiment;

[0046] Figure 6 is a schematic diagram of a model training method provided in this specific embodiment;

[0047] Figure 7 is a schematic diagram of an APP page change recognition method provided in this specific embodiment;

[0048] Figure 8 is a structural block diagram of the page change recognition device in this embodiment. Detailed Implementation

[0049] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0050] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning as understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these,” used in this application, do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to such processes, methods, products, or devices. The terms “connected,” “linked,” and “coupled,” used in this application, are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. The term “multiple” used in this application refers to two or more. The "and / or" operator describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: A alone, A and B simultaneously, and B alone. Typically, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," and "third," etc., used in this application are merely for distinguishing similar objects and do not represent a specific ordering of the objects.

[0051] The method embodiments provided in this example can be executed in a terminal, computer, or similar computing device. For example, running on a terminal, FIG1 is a hardware structure block diagram of the terminal for the page change recognition method of this application embodiment. As shown in FIG1, the terminal may include one or more (only one is shown in FIG1) processors 102 and a memory 104 for storing data, wherein the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA. The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that the structure shown in FIG1 is only illustrative and does not limit the structure of the terminal. For example, the terminal may also include more or fewer components than shown in FIG1, or have a different configuration than shown in FIG1.

[0052] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the page change recognition method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0053] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0054] The APP content testing project primarily involves testing hundreds of relevant apps to detect any inappropriate content on their pages. The general testing process involves automating operations on the app, acquiring the page text content and screenshots, and finally using a review model to review the text content and screenshots. Among the automated operations, determining whether the page has changed before and after a certain action is crucial. If this cannot be effectively determined, the automated operation may get stuck on a specific page and repeatedly acquire information from the same page. This is fatal in APP content testing, as it prevents the detection of new pages and places a significant burden on subsequent content review processes.

[0055] To determine whether an app's page has changed, the existing technologies mainly include the following methods.

[0056] The first method involves obtaining the page source code through the `driver.page_source` property of Appium, which is a string. By comparing the page source code before and after an Appium operation, it's possible to determine if the page has changed. However, this method is flawed for dynamic pages, as their source code is constantly changing. For example, a page might display the time, which is constantly changing. If Appium fails to execute an operation (such as clicking, swiping, or going back), judging whether the page has updated based on the source code will lead to a misjudgment. This could cause the system to mistakenly believe the operation was successful and fail to take effective measures to exit the current page, resulting in Appium being stuck on that page and unable to exit.

[0057] Another method involves taking screenshots of the app's page and comparing the hash similarity between the images before and after an Appium operation to determine if the page has changed. This method mitigates the interference from minor local changes in dynamic pages to some extent and is more reliable than the first method. However, this method has poor robustness and weak generalization ability, making it difficult to detect changes in dynamic pages with significant content variations. Furthermore, the method based on app page image hash similarity can cause automated operations to remain on pages with significant changes, preventing automatic navigation to the next page.

[0058] In addition, some methods determine whether a page has been updated by checking the existence of specific elements. However, this method is only effective for specific apps and specific pages; otherwise, it will fail when selecting dynamic page elements. Therefore, it lacks universality and is not suitable for scenarios involving testing hundreds of apps.

[0059] Therefore, this application provides a method for identifying page changes, applied to a terminal device, to identify changes in APP pages on mobile devices, thereby improving the accuracy and efficiency of page change identification. The terminal device includes servers, computers, or PCs, while mobile devices include mobile phones, tablets, etc.; one terminal device can control multiple mobile devices, or multiple terminal devices can control multiple mobile devices, without specific limitations.

[0060] This embodiment provides a method for identifying page changes. Figure 2 is a flowchart of the page change identification method provided in this embodiment. As shown in Figure 2, the method is applied to a terminal device, which is equivalent to the processor in Figure 1. The process includes the following steps:

[0061] Step S210: Obtain page training data from the mobile device, input the page training data into the preset first training model and second training model respectively for training, and obtain the test model; the initialization parameters of the first training model and the second training model are different.

[0062] The processor acquires page training data from the mobile device. Specifically, this page training data mainly includes screenshots of multiple apps on the mobile device. For example, this page training data can be obtained by taking screenshots of app pages using open-source tools such as Appium, or by acquiring page training data from different apps available online. Subsequently, the terminal device initializes a first and a second training model with different parameters using the page training data as input values, and obtains the outputs of the first and second training models. The loss of the model is then predicted based on the model's output, and the parameters of the first and second training models are updated to train the model. When the training model meets preset conditions, a test model is obtained. Further, the preset adjustment can be whether the accuracy of the training model meets preset conditions, or whether the number of iterations of the test model reaches a preset threshold; these are not specifically limited here.

[0063] Step S220: Input the page test data of the mobile device into the model to be tested to test the model and obtain the target recognition model; the first model to be trained, the second model to be trained and the model to be tested all include a target page attention module; the number of output channels of the convolution kernel in the target page attention module is determined by the number of input channels of the convolution kernel.

[0064] The processor acquires page test data from the mobile device using the same method as the page training data acquisition method described above. The difference between the page test data and the page training data lies in their intended use. After acquiring the page test data, the processor inputs it into the model under test for preprocessing. The preprocessed page test data is then input back into the model under test for testing. Test results are obtained, and the accuracy of the test data is determined based on these results. The model with accurate test results is then identified as the target recognition model. Furthermore, since both the training model and the test model include a target page attention module, the target recognition model obtained from these models also includes a target page attention module.

[0065] Furthermore, since information on an app page is generally arranged horizontally, both the training and testing models include a target page attention module that can integrate horizontal features. Simultaneously, to reduce computational cost, the output channels of the convolutional kernels need to be compressed. Specifically, when integrating horizontal features of the page, the number of output channels of the convolutional kernels is first compressed to half the number of input channels, followed by normalization and nonlinear operations. This improves the nonlinear fitting ability of the target page attention module.

[0066] Step S230: Input the page to be recognized into the target recognition model to identify whether the page has changed.

[0067] The processor acquires data of the page to be identified and inputs it into the target recognition model so that the model can identify whether the page has changed. Further, the page to be identified is the current page of the app. After inputting the page to be identified into the target recognition model, the model acquires the feature parameters of the page and determines whether these parameters match preset parameters. Based on these feature parameters, it then determines whether the page has changed. For example, the feature parameters of the page to be identified include, but are not limited to, channel parameters, width parameters, and height parameters.

[0068] Through the above steps, a parallel dual-model training model, including a target page attention module, is used to train the page training data to obtain the model to be tested. Then, the page test data is tested based on the model to be tested to obtain the target recognition model. Finally, page change recognition is performed based on the serial single target recognition model, which helps to improve the accuracy of page change recognition.

[0069] In some embodiments, step S210 includes steps S211 to S214.

[0070] Step S211: Obtain the attribute data of the page training data.

[0071] The processor assumes that the page in the training data changes. The changed page area in the training data can be rectangular, circular, or elliptical, without specific limitations. Furthermore, when the changed page area is rectangular, the attribute data of the page training data includes the width and height of the page.

[0072] Step S212: Preset the page change rate, and determine the page change area based on the page change rate and attribute data.

[0073] The page change rate is a randomly generated value within a preset range. For example, the page change rate can be generated using Python's standard library `random` module, JavaScript's `Math.random()` function, or C++'s... <random>The library is not specifically defined here. The processor calculates the page change area based on the page change rate and attribute data; further, it calculates the area, width, height, and coordinate values ​​of the page change area.

[0074] In one embodiment, the preset page change rate is updated based on the information of the page change area. Then, when iteratively training the model to be trained, the updated page change rate is used to determine the page change area, which helps to further improve the accuracy of the first model to be trained and the second model to be trained.

[0075] Step S213: Fill the changed area of ​​the page to obtain the first target page training data, and input the first target page training data into the first model to be trained.

[0076] Once the page change area is determined, it is filled; specifically, pixel values ​​are filled into the page change area. For example, when filling the page change area with pixel values, the filling can be done according to actual needs, including but not limited to filling with black pixels with a pixel value of 0, filling with white pixels with a pixel value of 255, or filling with other pixel values; a block of the same size can also be cropped from another page image and copied to the page change area. After filling the page change area, the first target page training data is obtained, and then the first page training data is input into the first model to be trained for training, obtaining the first training output data.

[0077] Step S214: Perform a deep copy of the page training data to obtain the second target page training data; input the second target page training data into the second model to be trained.

[0078] The processor performs a deep copy of the page training data to obtain the second target page training data. This second page training data is then input into the second model to be trained, yielding the second training output data. Furthermore, a deep copy of the page training data creates a second target page training data set with identical content but a different memory location. This provides independence and integrity to the page training data, facilitating the implementation of a parallel dual-model training method and further improving the accuracy and completeness of the training data.

[0079] Through the above steps, the processor processes the page training data in different ways to obtain the first target page training data and the second target page training data, which are then input into the first and second training models respectively for training. This facilitates the implementation of a parallel dual-model training method, further improves the accuracy and completeness of data training, and enhances the accuracy of page change recognition based on the aforementioned training models.

[0080] In some embodiments, step S210 includes steps S215 to S216.

[0081] Step S215: Input the first target page training data into the first model to be trained to obtain the first training output data, and input the second target page training data into the second model to be trained to obtain the second training output data.

[0082] The processor inputs the training data from the first target page and the training data from the second target page into two models to be trained, respectively, and obtains the first training output data and the second training output data. Furthermore, the model structures of the first and second models to be trained are completely identical, and their network and training configurations are also completely identical; only the initialization of the model parameters differs.

[0083] Step S216: Determine the cosine distance between the first training output data and the second training output data. Based on the cosine distance and the page change rate, determine the model loss. Update the first and second training models according to the model loss to obtain the model to be tested.

[0084] Specifically, after the processor acquires the first training output data and the second training output data, it calculates the cosine distance between the first training output data and the second training output data, and then determines the predicted model loss based on the cosine distance and the page change rate. Specifically, the model loss is determined based on the cosine distance and the Euclidean distance of the page change rate, and the parameters of the two models to be trained are updated based on the model loss to obtain the model to be tested.

[0085] Through the above steps, the first and second target page training data processed by the processor are used to train two models to be trained, respectively, to obtain first and second training output data. Then, based on the training output data and page change rate, the model loss is determined, and the parameters of the model to be trained are iteratively updated to obtain a test model whose accuracy and number of iterations are within a preset range, thereby improving the accuracy of determining the test model. Furthermore, the test model obtained from each training iteration is saved.

[0086] In some embodiments, step S220 includes steps S221 to S223.

[0087] Step S221: Preprocess the page test data according to the preset page change rate to obtain the first page test data and the second page test data.

[0088] Specifically, the processor preprocesses the page test data according to the method of obtaining the first target page training data and the second target page training data by preprocessing the page training data, thereby obtaining the first page test data and the second page test data. This provides independence and integrity for the page training data, which further helps to improve the accuracy and completeness of the data test.

[0089] Step S222: Input the first page test data and the second page test data into the model to be tested, respectively, to obtain the first page test results and the second page test results. Determine the cosine distance between the first page test results and the second page test results, and obtain the test similarity based on the cosine distance.

[0090] The processor tests the model under test using test data from the first page and the second page, obtaining the test results for the first and second pages. It then calculates the cosine distance between these test results, which is the test similarity value. Testing the same model under test with different test data and obtaining the test similarity between different test results is beneficial for subsequently determining the test accuracy of the model under test based on the test similarity.

[0091] Step S223: Determine the test accuracy of the model to be tested based on whether the difference between the page change rate and the test similarity exceeds the preset tolerance threshold range; determine the target recognition model based on the test accuracy.

[0092] The processor calculates the difference between the real-time page change rate and the test similarity, and presets a tolerance threshold range. It then determines whether the difference exceeds this threshold range to assess the accuracy of the page test results for the model under test, thereby improving the test accuracy. Furthermore, the model with the highest test accuracy is selected as the target recognition model.

[0093] Through the above steps, when testing the model to be tested, the processor first preprocesses the page test data to obtain two different types of page test data, which are then input into the model to be tested to obtain test results. Finally, the test similarity is determined based on the test results, and the target recognition model is obtained by combining the page change rate. This helps to improve the recognition accuracy and efficiency of the target recognition model.

[0094] In some embodiments, step S230 includes steps S231 to S232.

[0095] Step S231: Obtain the features of the page to be identified and determine whether the features are consistent with the preset initialization parameter features.

[0096] When the processor identifies the page to be identified using the target recognition model, it first determines the features of the page using the target recognition model; specifically, these features include the page's channel features, width features, and height features. Then, it checks whether the features of the page to be identified match the preset initialization parameter features. Generally, the preset initialization parameter features are None.

[0097] In step S232, if the features match, it is determined that the page to be identified has changed; if they do not match, the cosine distance between the feature and the initial parameter feature is determined, and the page is determined to have changed based on the cosine distance.

[0098] If they match, the current page to be identified is determined to be a brand new page, and the preset initialization parameter features are updated according to the current page to be identified. The current page of the APP is then captured again to obtain a new page to be identified. If they do not match, the cosine distance between the features of the page to be identified and the features of the initialization parameters, i.e., the feature similarity, is calculated to determine whether the feature similarity exceeds the preset page change rate threshold, thereby determining whether the page to be identified has changed.

[0099] Through the above steps, when identifying the page to be identified, the target recognition model extracts the features of the page to be identified, compares whether the features of the page to be identified are consistent with the preset initialization parameter features, and whether the feature similarity exceeds the preset page change rate threshold, thereby determining whether the page to be identified has changed, which helps to improve the accuracy and efficiency of page change recognition.

[0100] Furthermore, the preset initialization parameter features are updated based on the characteristics of the page to be identified.

[0101] The present embodiment will be described and explained below through specific examples.

[0102] This embodiment also provides a page change recognition method based on deep learning. Figure 3 is a flowchart of a page change recognition method based on deep learning provided in this embodiment. As shown in Figure 3, the process includes the following steps:

[0103] Step S301, data collection.

[0104] Specifically, Figure 4 is a schematic diagram of a hardware structure provided in this specific embodiment. Referring to Figure 4, the hardware structure includes a terminal device and a mobile device. The terminal device is connected to the mobile device via wired or wireless means. One terminal device can connect to multiple mobile devices, and multiple terminal devices can connect to multiple mobile devices. In one embodiment, the terminal device is a computer, and the mobile device is a mobile phone. The computer is connected to the mobile phone via a wired connection, i.e., a USB cable. The specific connection mode selected is a non-charging model. Simultaneously, software environments are built on both the computer and the mobile phone to allow page data to be collected and processed on the computer. The data collection process mainly involves the computer issuing action commands simulating human operation, such as clicking and swiping, and then the mobile phone automatically executes the operation commands to operate the corresponding APP. Text and image data are collected from each page of the APP. The entire process is fully automated and requires no manual intervention.

[0105] Furthermore, the data collection steps include steps S11 to S13.

[0106] Step S11: Collect screenshot images of the APP page.

[0107] The app page screenshot data is denoted as dataset_origin. This step is mainly used to collect page screenshots of the app to be tested in order to create a training dataset. Furthermore, open-source tools such as Appium can be used to take app page screenshots. Specifically, in Python, the `save_screenshot` method of Selenium can be used to save screenshots, and in Java, the `TakesScreenshot` interface can be used to take screenshots.

[0108] Step S12: Perform data cleaning on dataset_origin, removing duplicate or invalid images to obtain dataset_clean.

[0109] Step S13: Divide dataset_clean into three datasets: training dataset_train, evaluation dataset_eval, and test dataset_test. The ratio of the data in these three datasets is generally 8:1:1.

[0110] The training dataset includes the page training data in the aforementioned embodiments, and the test dataset includes the page test data in the aforementioned embodiments. After training the model to be trained using the page training data in the training dataset, the model obtained from each training is verified using the evaluation dataset, and the number of iterations (epoch) and the accuracy of the model on the evaluation dataset (dataset_eval) are recorded.

[0111] Step S302, data preprocessing.

[0112] Specifically, data preprocessing is used to obtain model input data and labels. The model input data here refers to the first target page training data, the second target page training data, the first page test data, and the second page test data in the aforementioned embodiments; the labels here refer to the page change rate in the aforementioned embodiments. The data preprocessing steps are as follows:

[0113] Step S21: Obtain an image and denote it as img.

[0114] Step S22: Obtain the width and height of the img element, denoted as w and h respectively. The width and height here are the attribute data in the aforementioned embodiment.

[0115] Step S23: Randomly generate a label with a value range of (0, 1). To simulate changes in the app page, such as banners, we assume that a region within the image has changed, and that this region is rectangular. Here, the label represents the proportion of the changed area of ​​the image to the total area of ​​the image.

[0116] Step S24, calculate the area of ​​the region where img changes:

[0117] S = label * w * h

[0118] Where label represents the proportion of the area of ​​the variable region of img to the total area of ​​img, w represents the width of img, and h represents the height of img.

[0119] Step S25: Obtain the width of the changing region of the image, denoted as w_v; where w_v = w if np.Sqrt(S) > welse random.uniform(np.Sqrt(s), w), the above formula expresses that the width of the changing region generated should be smaller than the width of the mobile device, i.e., the phone. Then, convert w_v to an integer, i.e., w_v = int(w_v).

[0120] Step S26, obtain the height of the changing region of img: h_v = int(s / w_v); the value of height h_v is the integer ratio of the area to the width of the changing region of img.

[0121] Step S27, update the value of label; that is, label = 1.0 * w_v * h_v / (w * h), where the value of label is the floating-point ratio of the area of ​​the changed region of img to the total area of ​​img.

[0122] Step S7: Determine the coordinates (x, y) of the top left corner of the changing region; where x = (w - w_v) * random.random(), y = (h - h_v) * random.random(), and random.random() is the method for obtaining random numbers in Python.

[0123] Step S28: Perform a deep copy of the image img to obtain img_c.

[0124] Step S29: Fill the changing areas of img with pixel values ​​and return img, img_c, label.

[0125] Furthermore, once the top-left corner coordinates (x, y) and width w_v and height h_v of the changing region are determined, the position of the changing region on the img is completely determined. At this point, the changing region can be filled. It can be filled entirely with 0 (i.e., the changing region is black), entirely with 255 (i.e., the changing region is white), or a specific pixel value. Alternatively, a block of the same size can be cropped from another image and copied to this region. Without loss of generality, we will use the method of filling with 0 for illustration, i.e., img[:, x:x+w_v, y:y+h_v] = numpy.zeros((w_v,h_v)). Furthermore, the features of the img mentioned above are (c,w,h), which correspond to the channels, width, and height of the img, respectively.

[0126] Step S303, Model Design.

[0127] Specifically, to balance performance and efficiency, the training model, testing model, and recognition model provided in this embodiment are all modified based on MobileNetV3. MobileNetV3 mainly includes an inverse residual module, an SE attention module, a basic convolutional module, and a classification head. The SE attention module can be enabled or disabled via the parameter `se`. If `se` is True, the SE attention module is enabled; if `se` is False, it is disabled. However, the SE channel attention module cannot capture the spatial information of APP page images, and the information on APP pages is generally arranged horizontally. Therefore, this embodiment uses a page attention structure (PAttention) to replace the SE channel attention module, while removing the MobileNetV3 classification head, leaving other structural components unchanged. This page attention structure is the target page attention module from the aforementioned embodiment.

[0128] Figure 5 is a structural diagram of the one-page attention structure provided in this specific embodiment. As shown in Figure 5, the attention module PAttention first integrates the horizontal features of the feature map of the current input data, and compresses the channels to half of their original size using a convolutional kernel conv1×3 to reduce computation. Then, it performs normalization using a Batch Normalization (BN) layer and nonlinear operations using a ReLU layer to improve the network's nonlinear fitting ability. Next, it reduces the number of channels in the feature map to 1 using a convolutional kernel conv1x1. Finally, the spatial attention of the feature map is obtained using the sigmoid function.

[0129] Furthermore, the convolution and its parameters used in conv1x3 are as follows:

[0130] nn.Conv2d(in, out, kernel_size=(1, 3), stride=1, padding=(0, 1))

[0131] Where in is the number of input channels, out is the number of output channels of the convolution kernel, out = int(0.5*in), where int represents an integer value; kernel_size is the size of the convolution kernel, kernel_size = (1, 3) means that the height of the convolution kernel is 1 and the width is 3, which means that the features in the horizontal direction are integrated; stride is the stride of the convolution kernel, and padding is the feature padding parameter; the channels of BN must be consistent with the output channels of conv, so the number of channels of BN is out.

[0132] Step S304, Model Training.

[0133] Specifically, Figure 6 is a schematic diagram of a model training method provided in this specific embodiment. Referring to Figure 6, firstly, an image is input and preprocessed to obtain image 1-img and image 2-img_c, where image 1 is the first target page training data and the first target page training data in the aforementioned embodiment. Then, image 1 and image 2 are input to model A and model B respectively to obtain training results embed_A and embed_B, and the cosine distance cos between embed_A and embed_B is calculated. The model prediction loss is calculated based on the cosine distance cos and the aforementioned label. Here, model A is the first training model in the aforementioned embodiment, and model B is the second training model in the aforementioned embodiment. Subsequently, model A and model B are iteratively updated using the model prediction loss until the number of iterations reaches a preset iteration threshold, or the accuracy of model A or model B on the evaluation dataset is within a preset range.

[0134] The specific steps are as follows:

[0135] Step S31: Input an image and denote it as img.

[0136] Step S32: Preprocess img according to the data preprocessing method in step S302 to obtain img, img_c and label.

[0137] Step S33: Randomly input img and img_c into model A and model B.

[0138] Model A and Model B have completely identical model structures, network configurations, and training configurations; the only difference is the initialization of model parameters. The network structures of Model A and Model B can be determined using the model design module in step S303. A brief summary of the steps is as follows:

[0139] Using MobileNetV3 as the base network; replacing the SE channel attention of MobileNetV3 with PAttention; configuring and enabling PAttention in the last 5 layers of MobileNetV3 in the same way as configuring SE channel attention in MobileNetV3; removing the classification head of MobileNetV3, the output is the feature.

[0140] Step S34: Obtain the output embed_A of model A and the output embed_B of model B;

[0141] Step S35, calculate the cosine distance d between embed_A and embed_B, that is:

[0142] d = cos(embed_A, embed_B)

[0143] Step S36: Calculate the model prediction loss.

[0144] The L2 loss function is used, which is the Euclidean distance between the cosine distance d and the label as the loss of the model prediction. Then, backpropagation updates the parameters of both models simultaneously.

[0145] Step S37: Iteratively train the model.

[0146] In this process, models A and B are trained using the training dataset dataset_train according to steps S31-S36, and the models obtained from each training are verified using the evaluation dataset dataset_eval. The number of iterations (epoch) and the accuracy (acc) of the model on dataset_eval are recorded. The models obtained from each training are saved as model_epoch, such as model1, model2, etc. When calculating the accuracy, the prediction error is considered to be correct as long as it is within a certain range.

[0147] Step S38: Determine whether the current epoch is greater than the preset threshold or whether the acc meets the early stopping condition; if yes, stop model training; if no, return to step S37.

[0148] Step S305, Model Testing.

[0149] Specifically, model testing is mainly used to select the best model for subsequent recognition of changes to the app's page; the steps are as follows:

[0150] Step S41: Load the model parameters using the modified MobileNetV3 to obtain the model. Further, this model can be modelA, modelB obtained in step S302, or any model from the iterative process.

[0151] Step S42: Input an image and denote it as img.

[0152] Step S43: Preprocess img according to the data preprocessing method in step S302 to obtain img, img_c and label.

[0153] Step S44: Input img into model to get embed_A, and input img_c into model to get embed_B.

[0154] Step S45: Calculate the similarity between the two images, i.e., sim = cos(embed_A, embed_B).

[0155] Step S46: If sim-label < p, the prediction is considered correct; otherwise, the prediction is considered incorrect. Here, parameter p is the tolerable range of the prediction. For example, parameter p is set to 0.01.

[0156] Step S47: Repeat steps S42-S46 using the test dataset dataset_test to obtain the test accuracy acc of the model.

[0157] Step S48: Repeat steps S41-S47 for all models A and B trained in step S304 to obtain the accuracy of all models. The model with the highest accuracy is denoted as model_best. Here, model_best is the target recognition model in the aforementioned embodiment.

[0158] Step S306: APP page change recognition.

[0159] Specifically, Figure 7 is a schematic diagram of an APP page change recognition method provided in this specific embodiment. Referring to Figure 7, the page change recognition method is as follows: First, initialize the parameter features, that is, preset the initialization parameter features to None, and preset the page change rate threshold p_var to 0.3; then input the current page of the APP, that is, the page to be recognized refers to the target recognition model, and obtain the feature embed; then determine whether the embed_pre corresponding to the image is None; if yes, then determine that the page where the current image is located is the changed page, and update the initialization parameter features embed_pre according to the image's feature embed; if no, then calculate the feature similarity sim between the image's feature embed and the initialization parameter features embed_pre, and determine the size of the feature similarity sim and the page change rate threshold p_var. If sim is not less than p_var, then determine that the page where the current image is located is the changed new page; if sim is less than p_var, then determine that the page where the current image is located is the unchanged old page. Then update the initialization parameter features embed_pre according to the image's feature embed; and re-acquire the next image, and perform page change recognition on the next image. The specific steps are as follows:

[0160] Step S51: Initialize parameters by setting embed_pre = None.

[0161] Step S52: Load the model_best model parameters using the modified mobileNetV3 to obtain the model.

[0162] Step S53: Take a screenshot of the current page of the APP to obtain the image img.

[0163] Step S54: Use model_best to extract features from img to obtain embed.

[0164] Step S55: Determine whether embed_pre is None.

[0165] If embed_pre is None, then the current page is considered a completely new page, and embed_pre = embed; proceed to step S51; if embed_pre is not None, then calculate the similarity between the current page and the previous page, i.e.:

[0166] sim = cos(embed, embed_pre)

[0167] embed_pre = embed

[0168] Here, `embed` represents the features of the page to be identified, and `embed_pre` represents the preset initialization parameter features. `embed_pre` is updated based on `embed` after the page to be identified has changed.

[0169] Step S56: Determine whether sim < p_var is true; if true, they are considered to be the same page; if false, they are considered to be different pages; where p_var is a hyperparameter representing the critical value of page change rate, which can be set to 0.3.

[0170] This embodiment uses a modified MobileNetV3 model to identify page change rates, which is more robust and has a higher accuracy in page change recognition than existing algorithms based on page source code and image hashing. During training, a data-parallel dual-model structure is used, while during recognition, a data-serial single-model structure is used, which improves the model training efficiency and accuracy, while reducing the cost and latency of model inference.

[0171] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0172] This embodiment also provides a page change recognition device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. The terms "module," "unit," "subunit," etc., used below can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0173] Figure 8 is a structural block diagram of the page change recognition device of this embodiment. As shown in Figure 8, the device includes a training module 10, a testing module 20 and a recognition module 30.

[0174] The training module 10 is used to acquire page training data from the mobile device, input the page training data into the preset first training model and second training model respectively for training, and obtain the test model; the initialization parameters of the first training model and the second training model are different.

[0175] The test module 20 is used to input page test data from the mobile device into the model to be tested, so as to test the model to be tested and obtain the target recognition model; the first model to be trained, the second model to be trained, and the model to be trained all include a target page attention module; the number of output channels of the convolution kernel in the target page attention module is determined by the number of input channels of the convolution kernel.

[0176] The recognition module 30 is used to input the page to be recognized into the target recognition model in order to identify whether the page has changed.

[0177] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0178] This embodiment also provides an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0179] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0180] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0181] S1, obtain page training data from the mobile device, input the page training data into the preset first training model and second training model respectively for training, and obtain the test model; the initialization parameters of the first training model and the second training model are different.

[0182] S2, input the page test data of the mobile device into the model to be tested to test the model and obtain the target recognition model; the first model to be trained, the second model to be trained and the test model to be trained all include a target page attention module; the number of output channels of the convolution kernel in the target page attention module is determined by the number of input channels of the convolution kernel.

[0183] S3, input the page to be recognized into the target recognition model to identify whether the page has changed.

[0184] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0185] Furthermore, in conjunction with the page change identification method provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the page change identification methods in the above embodiments.

[0186] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0187] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0188] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0189] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.< / random>

Claims

1. A method of recognizing a page change, characterized by, The method is applied to a terminal device and comprises: Obtaining page training data of a mobile terminal device, inputting the page training data into a preset first training model and a second training model respectively for training to obtain a test model; the first training model and the second training model have different initial parameters; Inputting page test data of the mobile terminal device into the test model to test the test model to obtain a target recognition model; the first training model, the second training model and the test model all comprise a target page attention module; the output channel number of a convolution kernel in the target page attention module is determined by the input channel number of the convolution kernel; Inputting a page to be recognized into the target recognition model to identify whether the page has changed.

2. The method of claim 1, wherein The obtaining of the page training data of the mobile terminal device, the inputting of the page training data into the preset first training model and the second training model respectively for training comprises: Obtaining attribute data of the page training data; Presetting a page change rate, determining a page change area according to the page change rate and the attribute data; Filling the page change area to obtain first target page training data and inputting the first target page training data into the first training model.

3. The method of claim 2, wherein The obtaining of the page training data of the mobile terminal device, the inputting of the page training data into the preset first training model and the second training model respectively for training comprises: Deeply copying the page training data to obtain second target page training data; Inputting the second target page training data into the second training model.

4. The method of claim 3, wherein The obtaining of the page training data of the mobile terminal device, the inputting of the page training data into the preset first training model and the second training model respectively for training to obtain a test model comprises: Inputting the first target page training data into the first training model to obtain first training output data and inputting the second target page training data into the second training model to obtain second training output data; Determining the cosine distance of the first training output data and the second training output data; According to the cosine distance and the page change rate, determining a model loss, updating the first training model and the second training model according to the model loss to obtain a test model.

5. The method of claim 1, wherein The inputting of the page test data of the mobile terminal device into the test model to test the test model to obtain a target recognition model comprises: Pretreating the page test data according to a preset page change rate to obtain first page test data and second page test data; Inputting the first page test data and the second page test data into the test model respectively to obtain first page test results and second page test results; Determining the cosine distance of the first page test results and the second page test results, obtaining a test similarity according to the cosine distance; According to whether the difference between the page change rate and the test similarity exceeds a preset tolerance threshold range, determining the test accuracy of the test model; According to the test accuracy, a target recognition model is determined.

6. The method of claim 1, wherein The input to be identified page is input to the target recognition model to identify whether the page has changed, comprising: Obtaining the features of the page to be identified, and determining whether the features are consistent with the preset initialization parameter features; If consistent, it is determined that the page to be identified has changed; if not consistent, the cosine distance between the features and the initialization parameter features is determined, and whether the page has changed is determined according to the cosine distance.

7. The method of claim 6, wherein The input to be identified page is input to the target recognition model to identify whether the page has changed, further comprising: According to the features of the page to be identified, the preset initialization parameter features are updated.

8. An apparatus for recognizing a page change, characterized by comprising: The device comprises a training module, a test module and an identification module. The training module is configured to obtain page training data of a mobile terminal device, input the page training data to a preset first to-be-trained model and a second to-be-trained model respectively for training to obtain a to-be-tested model, and the initialization parameters of the first to-be-trained model and the second to-be-trained model are different. The test module is configured to input page test data of the mobile terminal device to the to-be-tested model to test the to-be-tested model to obtain a target recognition model, the first to-be-trained model, the second to-be-trained model and the to-be-tested model all comprise a target page attention module, and the output channel number of the convolution kernel in the target page attention module is determined by the input channel number of the convolution kernel. The identification module is configured to input a page to be identified to the target recognition model to identify whether the page has changed. 9.An electronic device comprising a memory and a processor, the electronic device characterized by, The memory stores a computer program, and the processor is configured to run the computer program to execute the page change identification method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the page change identification method of any one of claims 1 to 7.

Images