APP response time analysis method based on deep neural network and distributed computing

Abstract

The invention discloses an APP response time analysis method based on a deep neural network and distributed computing, and belongs to the technical field of software testing. The method comprises the following steps: loading a video by recording an APP page, and splitting and preprocessing the video frame by frame by using a distributed CPU server to generate reasoning data; and the central GPU server performs reasoning based on the trained deep network model, judges the state of each frame, determines accurate starting and ending frames of page loading through a distributed post-processing mechanism, and accurately calculates the loading time perceived by a user. According to the method, the loading time perceived by the user is accurately reflected, and the method is suitable for cross-platform competitive product analysis; resource utilization is optimized through intelligent distributed calculation, large-scale video processing time is greatly shortened, meanwhile, an automatic decision-making mechanism is introduced, manual intervention cost is reduced, and full-automatic efficient performance testing and analysis are achieved.

Description

Technical Field

[0001] The present invention relates to a method for analyzing the response time of an APP, specifically a method for analyzing the response time of an APP based on a deep neural network and distributed computing, belonging to the technical field of software testing. Background Art

[0002] With the wide popularity of mobile applications (APPs) and the increasing requirements for user experience, their application performance has become a key factor affecting user retention rate and brand value. Poor-performing APPs not only lead to user loss and damage to brand reputation, but also directly affect commercial conversion rate and user satisfaction.

[0003] In the field of APP page loading performance testing, there are currently the following mainstream methods: Android Device Monitor. This method directly relies on the resources provided by development tools and is suitable for developers to quickly evaluate the startup time. However, in actual use, data deviation may occur due to factors such as system load.

[0004] ADB Shell commands. This method is intuitive and easy to use, but the output time only represents the recording time point of the system log, and there is a significant difference from the visually perceived loading completion time of users, especially in the case of dynamic content loading scenarios, where the error is more obvious.

[0005] Code instrumentation. Although this method is simple to implement and can capture accurate internal execution times, it has three major limitations: 1) It is only applicable to self-developed APPs with access to source code and cannot be used for competitive product analysis; 2) It is often difficult to accurately judge the fully loaded state of complex UI elements, especially in asynchronous loading and dynamic rendering scenarios; 3) The instrumentation process itself will have an additional overhead on application performance, affecting the measurement accuracy. Therefore, its application scope is limited and it is difficult to meet the comprehensive performance evaluation requirements.

[0006] High-speed camera or screen recording analysis. Traditional pixel-based analysis methods face severe challenges when dealing with dynamic pages: 1) It is difficult to determine the optimal comparison threshold, resulting in inaccurate judgments; 2) The computational complexity is high, and the efficiency is extremely low when processing a large number of video files; 3) It lacks adaptability to applications with different visual styles and requires frequent manual intervention to adjust parameters. Summary of the Invention

[0007] Object of the Invention: Aiming at the above problems, the object of the present invention is to provide a method for analyzing the response time of an APP based on a deep neural network and distributed computing, which splits the preprocessing and postprocessing parts and distributes them to different computing nodes for processing to achieve more accurate, efficient, and automated APP performance analysis.

[0008] Technical solution: On the one hand, the APP response time analysis method based on deep neural network and distributed computing provided by the present invention includes the following steps: Record the video of the APP page loading and upload it to the file server; Use the distributed CPU server to obtain the video from the file server, split the video into frame-by-frame pictures, preprocess the pictures, generate a data format file for classification model inference, and upload the data format file to the file server; Obtain the arrival timestamp information of each frame of the image, associate the arrival timestamp information with the corresponding image data; store the associated arrival timestamp information in the local storage medium of the distributed CPU server in a structured data format; Use the central GPU server to receive the analysis task request of the APP response time, obtain the preprocessed data format file from the file server, use the trained classification model to perform inference on the data format file, and upload the inference result to each distributed CPU server; Use the distributed CPU server to post-process the received inference result, obtain the start frame and end frame of each page loading, calculate the page loading time, and at the same time obtain the start screenshot and end screenshot during each page loading; Use the central GPU server to summarize the analysis results of the distributed CPU server, store the analysis results in the database, and provide an interface for query.

[0009] Further, before the step of using the trained classification model to perform inference on the data format file includes: Use the data set and labels to train the classification model.

[0010] Further, the step of using the data set and labels to train the classification model includes: Construct a data set, where the data set is composed of the disassembled videos of the APP page loading, and each frame of the video is divided into different page states, including stable state, loading state and click state pictures, and the page states constitute different class labels; Construct a classification model, where the classification model is a deep neural network; Construct a loss function; Use the pictures in the constructed data set as the input of the classification model, the output of the classification model is the class probability score corresponding to each picture, use the loss function to measure the difference between the model prediction and the actual label, and use gradient descent to optimize the classification model.

[0011] Further, the step of using the distributed CPU server to post-process the received inference result includes: Obtain the start frame numbers and end frame numbers of all click states, and record them as a list , represents the i-th tuple. The list contains n tuples, and each tuple is denoted as , and in each tuple represents the frame number at which the i-th click state starts to click, represents the frame number at which the i-th click state releases the click; Traverse each tuple in the list in sequence, obtain the count of different state frames between the end frame of the current click state and the start frame of the next click state, select the loading state with the largest number of occurrences and at least 3 frames, and use this loading state as the loading state triggered by the current click operation; if not found, skip the current tuple, traverse the next tuple, and calculate the start frame number and end frame number at which the current state appears after finding; Regard all blank loading states as the same class, and reset the start frame of the loading state to the next frame of the end frame of the current click state; Summarize the start frame numbers and end frame numbers of all loading states, query the arrival times of the start frame and end frame using the timestamp information file, calculate the actual page loading time, and obtain the pictures at the start and end of each page loading.

[0012] Furthermore, the steps for preprocessing the pictures include: Use an image processing library to read the video and obtain each frame of the picture frame by frame; Perform a scaling operation on each frame of the picture; Perform a normalization operation on the scaled picture; Perform standardization processing with the mean and standard deviation; Stitch the processed frame data; Perform lossless compression on the stitched data and store the compressed data in a structured format.

[0013] On the other hand, the APP response time analysis system based on deep neural network and distributed computing provided by the present invention includes: A file server for storing the original video file, the preprocessed data, and the processing results; Multiple distributed CPU servers for preprocessing the video file and post-processing the inference results of the classification model; A central GPU server for creating analysis tasks, performing classification model inference, summarizing the processing results, and providing a query interface.

[0014] Furthermore, the system further includes a task scheduling module for coordinating the communication and data transmission among the distributed CPU servers, the central GPU server, and the file server.

[0015] Furthermore, the system adopts an asynchronous communication mechanism.

[0016] Furthermore, the system adopts a fault tolerance mechanism.

[0017] Beneficial effects: Compared with the prior art, the significant advantages of the present invention are as follows: 1. Improve the accuracy of APP performance analysis: The present invention uses a deep neural network for frame-by-frame analysis, which can more truly reflect the user's perception experience, thereby improving the accuracy of page loading time measurement, meaning that it is not limited to its own application programs, but can also objectively analyze the APPs of competitors; 2. Enhance processing efficiency: The present invention adopts a distributed computing method, which can distribute the preprocessing and postprocessing tasks to different computing nodes, make full use of the computing resources of the server, and speed up the data processing speed; and the ability of multi-node parallel processing of videos greatly shortens the overall analysis time, meeting the needs of large-scale data processing; 3. Save human resource investment: The present invention can automatically process complex data analysis tasks, reduce the need for manual intervention, thereby reducing the labor cost of testing and analysis. This automated analysis method not only improves the work efficiency of the team, but also can more quickly discover performance problems during the testing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is the overall flow diagram of the method in the embodiment of the present invention; Figure 2 is the UML sequence diagram of distributed computing in the embodiment of the present invention; Figure 3 is the flowchart of preprocessing the video in the embodiment of the present invention; Figure 4 is the flowchart of postprocessing the model inference result in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0020] Embodiment 1 The method for analyzing the APP response time based on a deep neural network and distributed computing described in this embodiment has a flowchart as Figure 1 shown, combined with Figure 2 the UML sequence diagram of distributed computing shown, the method includes the following steps: Step 1, record the video of the APP page loading and upload it to the file server.

[0021] Step 2: Use the distributed CPU servers to obtain the video from the file server, split the video into frame-by-frame pictures, preprocess the pictures to generate a data format file for classification model inference, and upload the data format file to the file server.

[0022] The number of the distributed CPU servers can be dynamically adjusted according to the processing requirements. Load balancing is achieved through a task queue to ensure efficient processing of a large number of video files. When a certain distributed CPU server fails to process, the system can automatically transfer the task to other available distributed CPU servers to ensure the continuity and reliability of the analysis task. Use the stress testing server to record the video and upload the video file to the file server. Download the video stored in the file server to the distributed CPU servers, use an image processing library to read the video, and then split the video into frame-by-frame pictures, obtaining each frame picture one by one. The image processing library includes but is not limited to OpenCV, FFmpeg, etc.

[0023] Combined Figure 3 As shown, further, the steps for preprocessing the pictures include: Use an image processing library such as OpenCV to read the video and obtain each frame picture one by one; Perform a scaling operation on each frame picture. For example, it can be uniformly scaled to the size of; Perform a normalization operation on the scaled picture so that its pixel values are between 0 and 1; Perform standardization processing with a mean and standard deviation of 0.5, for example; Stitch the processed frame data in batches of 256; Use the DEFLATE algorithm to perform lossless compression on the stitched data, and store the compressed data in a structured format, such as storing it as an npz file.

[0024] Step 3: Obtain the arrival timestamp information of each frame of image, associate the arrival timestamp information with the corresponding image data; store the associated arrival timestamp information in a structured data format in the local storage medium of the distributed CPU servers.

[0025] The arrival timestamp of each frame of image can be obtained using the FFmpeg image processing library and saved as a JSON file stored locally on each distributed CPU server.

[0026] Step 4: Use the central GPU server to receive the analysis task request for the APP response time, obtain the preprocessed data format file from the file server, perform inference on the data format file using the trained classification model, and upload the inference result to each distributed CPU server.

[0027] Dataset Preparation: To enhance the classification model's ability to judge the starting frame of page loading, it is necessary to enable the pointer position in the developer mode on the mobile phone. After triggering a click operation, the touch point and its coordinates will be displayed on the screen, providing accurate visual features for the classification model to make judgments.

[0028] Furthermore, before the step of using the trained classification model to infer the data format file, it includes: Training the classification model using the dataset and labels.

[0029] Furthermore, the steps of training the classification model using the dataset and labels include: Construct a dataset, where the dataset consists of videos obtained by disassembling APP page loading. Each frame of the video is divided into different page states, including stable state, loading state, and click state pictures. The page states constitute different class labels; Construct a classification model, where the classification model is a deep neural network; Construct a loss function; Use the pictures in the constructed dataset as the input of the classification model. The output of the classification model is the class probability score corresponding to each picture. Use the loss function to measure the difference between the model prediction and the actual label, and use gradient descent to optimize the classification model.

[0030] In one example, the deep neural network can adopt the ResNet18 network model. The ResNet18 network model is a deep residual network. Due to its efficient classification ability and low computational overhead, it becomes an ideal choice for classification tasks. The cross-entropy loss function is used to measure the difference between the model prediction and the actual label. This is a loss function commonly used in multi-class classification problems. The optimizer adopts Adam, which is an adaptive learning rate optimization algorithm that can efficiently perform gradient descent optimization and quickly converge to the optimal solution.

[0031] Since there may be large blank areas during page loading, and the blank pages of different pages may be extremely similar, it is necessary to uniformly classify all blank loading pages into the same category to reduce classification errors. The training dataset is obtained by disassembling APP page loading videos. Then, the video frames are manually divided into different page states, including stable state, loading state, and click state images, to form different class annotations.

[0032] After the classification model inference is executed, the category to which each frame of the video belongs will be obtained. According to the classification results of the classification model, all states can be summarized into three categories: loading state, stable state, and click state. The specific categories included in each major category are shown in Table 1.

[0033] Table 1

[0034] Step 5: Use the distributed CPU server to post - process the received inference results to obtain the start frame and end frame of each page load, calculate the page load time, and simultaneously obtain the start screenshot and end screenshot during each page load.

[0035] Combined with Figure 4 , further, the steps of using the distributed CPU server to post - process the received inference results include: Obtain the start frame numbers and end frame numbers of all click states, denoted as list , represents the i - th tuple, and the list includes n tuples, each tuple is denoted as , in each tuple represents the frame number when the i - th click state starts clicking, represents the frame number when the i - th click state releases the click; Traverse each tuple in the list in sequence, obtain the count of different state frames between the end frame of the current click state and the start frame of the next click state, select the load state with the largest number of occurrences and at least 3 frames, and use this load state as the load state triggered by the current click operation; if not found, skip the current tuple, traverse the next tuple, and calculate the start frame number and end frame number when the current state appears after finding it; Treat all blank load states as the same class, and reset the start frame of the load state to the next frame of the end frame of the current click state; Summarize the start frame numbers and end frame numbers of all load states, query the arrival times of the start frame and end frame using the timestamp JSON file, calculate the actual page load time, and obtain the pictures when each page starts loading and ends loading.

[0036] In one example, first, all the start and end frame numbers of the click states are obtained, such as [(10, 20), (30, 40), (60, 70)]. The list contains 3 tuples, and each tuple represents the frame number when a click state starts and the frame number when the click is released. Traverse each tuple in turn to obtain the count of different state frames between the end frame of the current click state and the start frame of the next click state. Select the loading state with the largest number of occurrences and at least 3 frames as the loading state triggered by the current click operation. If not found, skip to the next tuple loop. After finding it, calculate the start and end frames when the current state appears. Since there may be cases such as model classification errors, there may be multiple segments of loading states between two click frames. At the same time, some pages are blank when loading, and it is impossible to distinguish which loading state it is. During training, all blank loading states are regarded as the same class. Therefore, multiple loading states of the same class and blank loading states are merged. And because there is no change in the page within a few frames before and after the user triggers a click operation, most of the frames after the start frame of the click state are classified as the previous state of the click state. Therefore, the start frame of the loading state is reset to the next frame of the end frame of the current click state. Finally, summarize the start frame numbers and end frame numbers of all loading states, query the arrival times of the start frame and end frame using the timestamp JSON file, calculate the actual page loading time, and obtain the pictures when each page starts loading and ends loading.

[0037] Step 6: Use the central GPU server to aggregate the analysis results of the distributed CPU servers, store the analysis results in the database, and provide an interface for querying.

[0038] Use the central GPU server to aggregate the analysis results of the distributed CPU servers, store the analysis results in the database, and provide an interface for querying. The analysis results include the name, start frame number, start frame screenshot, start frame timestamp, end frame number, end frame screenshot, end frame timestamp, and loading time of each loading state of the APP.

[0039] In summary, this example provides an APP response time analysis method based on deep neural networks and distributed computing. By using deep neural networks for frame-by-frame analysis, this method can more realistically reflect the user's perception experience, thereby improving the accuracy of page loading time measurement. This means that it is not limited to its own application programs, but can also objectively analyze the APPs of competitors; adopting a distributed computing strategy, the preprocessing and postprocessing tasks can be assigned to different computing nodes, making full use of the computing resources of the servers and accelerating the data processing speed. The ability of multi-node parallel processing of videos greatly shortens the overall analysis time and meets the needs of large-scale data processing; it can automatically process complex data analysis tasks, reduce the need for manual intervention, and thus reduce the labor cost of testing and analysis.

[0040] Example Two The APP response time analysis system based on deep neural network and distributed computing described in this embodiment includes: A file server for storing original video files, preprocessed data, and processing results; Multiple distributed CPU servers for preprocessing video files and post-processing model inference results; A central GPU server for creating analysis tasks, performing classification model inference, summarizing processing results, and providing a query interface.

[0041] Furthermore, the system also includes a task scheduling module for coordinating communication and data transmission among the distributed CPU servers, the central GPU server, and the file server to ensure that each component works together according to a predefined timing process.

[0042] Furthermore, the system adopts an asynchronous communication mechanism, enabling each server component to process tasks in parallel and improving the overall throughput and performance of the system.

[0043] Furthermore, the system adopts a fault tolerance mechanism. When a certain distributed CPU server fails to process, the system can automatically transfer the task to other available servers to ensure the continuity and reliability of the analysis task.

[0044] The system uses distributed computing to accelerate the data processing speed and utilizes the data flow of distributed computing, including: a) The stress testing server is responsible for recording videos and uploading the video files to the file server; b) The file server is responsible for storing original video files, preprocessed data, and processing results; c) The distributed CPU servers are responsible for preprocessing video files, receiving and responding to asynchronous processing requests, and post-processing model inference results; d) The central GPU server is responsible for creating analysis tasks, performing deep neural network model inference, summarizing processing results, and providing a query interface; Furthermore, the system realizes distributed computing through the following timing process: S1, The stress testing server uploads the recorded video file to the file server; S2, The file server downloads the video file to the distributed CPU servers; S3, The stress testing server requests an asynchronous processing interface from the distributed CPU servers; S4, The distributed CPU servers respond to the asynchronous processing interface callback; S5, The distributed CPU servers upload the preprocessed result files to the file server; S6, The central GPU server downloads the preprocessed file from the file server; S7, The stress testing server creates an analysis task for the central GPU server; S8, The central GPU server returns the model inference result to the distributed CPU server; S9, The distributed CPU server returns the post-processing result to the central GPU server; S10, The stress testing server queries the analysis task result from the central GPU server.

Claims

1. An APP response time analysis method based on deep neural network and distributed computing is characterized by: The following steps are involved: Record the video of the APP page loading and upload it to the file server; Use the distributed CPU server to obtain the video from the file server, split the video into frame-by-frame images, pre-process the images, generate a data format file for classification model inference, and upload the data format file to the file server; Obtaining the arrival timestamp information of each frame of image, and associating the arrival timestamp information with the corresponding image data; The associated arrival timestamp information is stored in a structured data format in a local storage medium of a distributed CPU server; The central GPU server receives the analysis task request of the APP response time, obtains the pre-processed data format file from the file server, uses the trained classification model to infer the data format file, and uploads the inference result to each distributed CPU server; Use the distributed CPU server to post-process the received inference results, obtain the start frame and end frame of each page loading, calculate the page loading time, and obtain the start screenshot and end screenshot of each page loading process; The central GPU server is used to aggregate the analysis results of the distributed CPU servers, store the analysis results in the database, and provide interface queries.

2. The APP response time analysis method based on deep neural network and distributed computing according to claim 1 is characterized in that: The steps for using the trained classification model to infer the data format file include: Train the classification model using the dataset and labels.

3. The APP response time analysis method based on deep neural network and distributed computing according to claim 2 is characterized in that: The steps to train a classification model using a dataset and labels include: Construct a dataset consisting of videos of app page loading. Each frame of the video is divided into different page states, including stable state, loading state, and click state images. The page states constitute different category labels. Construct a classification model, where the classification model is a deep neural network; Construct loss function; The images in the constructed dataset are used as the input of the classification model. The output of the classification model is the category probability score corresponding to each image. The loss function is used to measure the difference between the model prediction and the actual label, and the classification model is optimized using gradient descent.

4. The APP response time analysis method based on deep neural network and distributed computing according to any one of claims 1 to 3 is characterized in that: The steps of post-processing the received inference results using the distributed CPU server include: Get the start and end frames of all click states and record them as a list , Represents the i-th tuple, the list includes n tuples, each tuple is recorded as , in each tuple Indicates the frame number of the i-th click state starting click, Indicates the number of frames when the i-th click state releases the click; Traverse each tuple in the list in turn, obtain the frame counts of different states between the end frame of the current click state and the start frame of the next click state, select the loading state with the largest number of occurrences and at least 3 frames, and use this loading state as the loading state triggered by the current click operation; if not found, skip the current tuple, traverse the next tuple, and calculate the start frame number and end frame number of the current state after finding it; Treat all blank loading states as the same category, and reset the start frame of the loading state to the next frame of the end frame of the current click state; Summarize the start and end frame numbers of all loading states, use the timestamp information file to query the arrival time of the start and end frames, calculate the actual page loading time and obtain the pictures when each page starts and ends loading.

5. The APP response time analysis method based on deep neural network and distributed computing according to any one of claims 1 to 3, characterized in that: The steps for preprocessing the image include: Use the image processing library to read the video and obtain each frame of the picture frame by frame; Perform scaling operations on each frame of the image; Normalize the scaled image; Standardization was performed using mean and standard deviation; Splicing the processed frame data; The concatenated data is losslessly compressed and stored in a structured format.

6. APP response time analysis system based on deep neural network and distributed computing, characterized by: include: File server, used to store original video files, pre-processed data and processing results; Multiple distributed CPU servers for preprocessing video files and post-processing classification model inference results; The central GPU server is used to create analysis tasks, perform classification model reasoning, summarize processing results, and provide query interfaces.

7. The APP response time analysis system based on deep neural network and distributed computing according to claim 6 is characterized in that: The system also includes a task scheduling module for coordinating communications and data transmission among the distributed CPU servers, the central GPU server, and the file server.

8. The APP response time analysis system based on deep neural network and distributed computing according to claim 7 is characterized in that: The system adopts an asynchronous communication mechanism.

9. The APP response time analysis system based on deep neural network and distributed computing according to claim 7 or 8, characterized in that: The system adopts a fault-tolerant mechanism.

Images