Application performance testing method, device, equipment and medium

Abstract

The present disclosure relates to an application performance testing method, device, equipment and medium. The method includes: obtaining a video file generated for executing a test action for an application; extracting features from the video file to obtain optical flow features of the video file; detecting the optical flow features of the video file through a pre-trained key frame detection model to obtain the relative position of the start frame and the relative position of the end frame in the video file that responds to the detection operation; and determining the response time of the application based on the relative position of the start frame and the relative position of the end frame and the frame rate of the video file. The embodiment of the present disclosure detects the optical flow features of the video file through a key frame detection model to obtain the relative position of the start frame and the relative position of the end frame in the video file that responds to the detection operation, and then determines the response time of the application, which can shield the influence of meaningless interface changes and irrelevant pictures, has a strong adaptability to changes in the interface content of the application, and is more applicable.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an application performance testing method, device, equipment and medium. Background Art

[0002] During the release and use of the application, the performance of the application will be evaluated and analyzed to clarify the direction for subsequent product optimization of the application. At present, the key frame annotation method based on video recording is usually used to record the operation process of the application into a video, and then the frame-by-frame calculation and classification method is used to calculate the peak signal-to-noise ratio, mean square error and other objective indicators frame by frame. Based on these objective indicators, the change and stability stages in the video are distinguished, and then the machine learning method is used to classify the images of different stages, mark the start and end stages, and finally deduce the start frame and end frame of the application's response to the user's operation to calculate the response time of the application to the user's operation.

[0003] Currently in practical applications, due to the increased complexity of the application scenarios themselves, classification errors are large at the beginning or end stages, or even lost, leading to detection failure. More importantly, frame-by-frame calculations are inefficient, making it difficult to meet current demands for efficiency and accuracy in application performance testing. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides an application performance testing method, apparatus, device and medium.

[0005] The present disclosure provides an application performance testing method, the method comprising:

[0006] Acquire a video file generated by executing a test action on an application; the video file can show the interface changes of the application in response to the test operation;

[0007] Extracting features from the video file to obtain optical flow features of the video file;

[0008] The optical flow features of the video file are detected by a pre-trained key frame detection model to obtain the relative position of the start frame and the relative position of the end frame in the video file that responds to the detection operation; wherein the key frame detection model is a deep convolutional network, and takes the optical flow features of the video file to be tested as input, and takes the relative position of the start frame and the relative position of the end frame as output;

[0009] The response time of the application is determined according to the relative position of the start frame, the relative position of the end frame and the frame rate of the video file.

[0010] In some embodiments, the key frame detection model is a deep convolutional model with cross-layer connections; wherein the cross-layer connections are used to connect and splice the results of convolutional layer outputs of different depths to fuse convolutional layer features of different time and space scales.

[0011] In some embodiments, the deep convolution model includes: a plurality of convolutional layers, a plurality of maximum downsampling layers, a plurality of adaptive downsampling layers, a splicing layer, and a plurality of fully connected layers; wherein the number of the plurality of convolutional layers is one more than the number of the plurality of maximum downsampling layers;

[0012] The output of one of the multiple convolutional layers is connected to an adaptive downsampling layer; the output of each of the remaining convolutional layers is connected to a maximum downsampling layer, and the output of each maximum downsampling layer is connected to a convolutional layer of the next depth; and the depth of the convolutional layer connected to the adaptive downsampling layer is greater than the depth of each convolutional layer connected to the maximum downsampling layer;

[0013] The output of each convolutional layer connected to the maximum downsampling layer is cross-connected with an adaptive downsampling layer;

[0014] The outputs of the multiple adaptive downsampling layers are all connected to the splicing layer;

[0015] The output of the splicing layer is connected to a fully connected layer, and the multiple fully connected layers are connected in series, and the output of the last fully connected layer connected in series is the relative position of the start frame and the relative position of the end frame in the video file that responds to the detection operation.

[0016] In some embodiments, the deep convolution model includes: 4 convolutional layers, 3 maximum downsampling layers, 4 adaptive downsampling layers, 1 splicing layer and 2 fully connected layers;

[0017] The first convolution layer is used to perform convolution processing on the optical flow features of the video file, the output of the first convolution layer is connected to the first maximum downsampling layer, and the output of the first convolution layer is cross-layer connected to the first adaptive downsampling layer; the output of the first maximum downsampling layer is connected to the second convolution layer;

[0018] The output of the second convolutional layer is connected to the second maximum downsampling layer, and the output of the second convolutional layer is cross-layer connected to the second adaptive downsampling layer; the output of the second maximum downsampling layer is connected to the third convolutional layer;

[0019] The output of the third convolutional layer is connected to the third maximum downsampling layer, and the output of the third convolutional layer is cross-layer connected to the third adaptive downsampling layer; the output of the third maximum downsampling layer is connected to the fourth convolutional layer;

[0020] The output of the fourth convolutional layer is connected to the fourth adaptive downsampling layer;

[0021] The output of the first adaptive downsampling layer, the output of the second adaptive downsampling layer, the output of the third adaptive downsampling layer and the output of the fourth adaptive downsampling layer are all connected to the splicing layer;

[0022] The output of the concatenated layer is connected to the first fully connected layer; the output of the first fully connected layer is connected to the second fully connected layer;

[0023] The output of the second fully connected layer is the relative position of the start frame and the relative position of the end frame in the video file that responds to the detection operation.

[0024] In some embodiments, determining the response time of the application according to the relative position of the start frame, the relative position of the end frame and the frame rate of the video file includes:

[0025] Determine the number of frames of the video file based on the duration and frame rate of the video file;

[0026] Determine the starting frame position based on the starting frame relative position and the number of frames of the video file;

[0027] Determining an end frame position based on the end frame relative position and the number of frames of the video file;

[0028] Based on the start frame position and the end frame position, a time consumption of the application program in responding to the detection operation is determined.

[0029] In some embodiments, the starting frame position is y1×N; the ending frame position is y2×N; wherein y1 is the relative position of the starting frame, and y1∈[0,1]; y2 is the relative position of the ending frame, and y2∈[0,1]; and N is the number of frames of the video file.

[0030] In some embodiments, determining the time taken for the application to respond to the detection operation based on the start frame position and the end frame position includes:

[0031] Determine the duration between two adjacent frames based on the frame rate of the video file;

[0032] Based on the start frame position and the end frame position, determining the number of frames included between the start frame and the end frame;

[0033] The time taken for the application to respond to the detection operation is determined based on the duration between two adjacent frames and the number of frames between the start frame and the end frame.

[0034] In some embodiments, the frame rate of the video file is a fixed frame rate.

[0035] The present disclosure also provides an application performance testing device, the device comprising:

[0036] An acquisition unit, used to acquire a video file generated by executing a test action on an application; the video file can show the interface changes of the application in response to the test operation;

[0037] An extraction unit, used for performing feature extraction on the video file to obtain an optical flow feature of the video file;

[0038] A first determination unit is used to detect the optical flow features of the video file through a pre-trained key frame detection model to obtain the relative position of the start frame and the relative position of the end frame in the video file that responds to the detection operation; wherein the key frame detection model is a deep convolutional network, and takes the optical flow features of the video file to be tested as input, and takes the relative position of the start frame and the relative position of the end frame as output;

[0039] The second determining unit is used to determine the response time of the application according to the relative position of the start frame, the relative position of the end frame and the frame rate of the video file.

[0040] In some embodiments, the key frame detection model is a deep convolutional model with cross-layer connections; wherein the cross-layer connections are used to connect and splice the results of convolutional layer outputs of different depths to fuse convolutional layer features of different time and space scales.

[0041] The present disclosure also provides an electronic device, which includes: a processor and a memory; the processor is used to execute the steps of any of the above methods by calling a program or instruction stored in the memory.

[0042] The present disclosure also provides a non-transitory computer-readable storage medium, which stores a program or instruction, wherein the program or instruction enables a computer to execute the steps of any of the above methods.

[0043] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0044] In the application performance testing method provided by the embodiment of the present disclosure, by obtaining a video file generated by executing a test action for the application; the video file can be feature extracted to obtain the optical flow feature of the video file; and then the optical flow feature is detected by the key frame detection model, and the relative position of the start frame and the relative position of the end frame in the video file that responds to the detection operation are output; thereby, the response time of the application is determined according to the relative position of the start frame and the relative position of the end frame and the frame rate of the video file. Therefore, the relative position of the start frame and the relative position of the end frame can be determined based on the optical flow feature, rather than directly based on the frame-by-frame image feature itself, thereby shielding the influence of meaningless interface changes and irrelevant pictures, and having a strong adaptability to the interface content changes of the application, so that the method has strong applicability; at the same time, the relative position of the start frame and the relative position of the end frame are determined based on the optical flow feature, rather than based on specific markers or stage classification, which improves the efficiency and accuracy of determining the key frame (i.e., the start frame and the end frame), thereby improving the problem of low detection efficiency; in addition, due to the strong applicability and high robustness, the probability of successful detection is increased, and the proportion of manual review is reduced, thereby reducing the labor cost of manual review after detection, and further improving the detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 A flowchart of an application performance testing method provided by an embodiment of the present disclosure;

[0048] Figure 2 A schematic diagram of the principle of optical flow feature extraction in the method of an embodiment of the present disclosure;

[0049] Figure 3 A schematic diagram of a key frame determination method in a method according to an embodiment of the present disclosure;

[0050] Figure 4 A schematic diagram of a deep convolution model in the method of an embodiment of the present disclosure;

[0051] Figure 5 A schematic diagram for comparing the method provided by the embodiment of the present disclosure with the related art;

[0052] Figure 6 A schematic diagram of the structure of an application performance testing device provided by an embodiment of the present disclosure;

[0053] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0054] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0055] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0056] The application performance testing method provided by the embodiment of the present disclosure is implemented based on optical flow feature extraction and key frame detection model (such as deep convolution model). Among them, the method extracts features from images based on optical flow, rather than directly based on the frame-by-frame image features themselves, thereby shielding the impact of meaningless interface changes and irrelevant pictures, and has a strong adaptability to changes in APP interface content, stronger applicability, stronger compatibility, and can greatly reduce the cost of manual calibration, which is conducive to improving detection efficiency; based on the change of optical flow, rather than based on specific markers or stages for classification, the problem of reduced detection efficiency is further improved; based on the above, stronger applicability and robustness reduce the labor cost of manual review after detection, and further improve detection efficiency. At the same time, the method uses a deep convolution model to directly calculate the position of the key frame, rather than calculating the objective indicators frame by frame and then classifying. Due to the high calculation efficiency of the model, it is faster to directly determine the key frame position based on the deep convolution model than the traditional method based on frame-by-frame calculation of PSNR (Peak Signal to Noise Ratio, peak signal-to-noise ratio), thereby also making the detection efficiency higher.

[0057] Combine the following Figures 1 to 7 The application performance testing method, apparatus, device and medium provided in the embodiments of the present disclosure are exemplified.

[0058] For example, Figure 1 A flowchart of an application performance testing method provided by an embodiment of the present disclosure. Figure 1 , the method may include:

[0059] S101. Obtain a video file generated by executing a test action on an application; the video file can display interface changes of the application in response to the test operation.

[0060] The video file is a video file obtained by recording the interface changes during the application program response operation. The video file includes key frames for determining the time taken for the application program response, which are shown as the start frame and the end frame respectively below.

[0061] For example, the video file can be recorded using an external camera, which will not occupy system resources, thereby not affecting the response time of the application under test, and making the performance index extraction more accurate. Alternatively, the video file can be recorded using a screen recording tool built into the terminal, for example, a terminal using an Android system can record the screen by using an Android debug bridge (adb) command built into the APP, so that there is no need to set up an additional camera, thereby simplifying the hardware equipment required.

[0062] S102: Extract features from the video file to obtain optical flow features of the video file.

[0063] Among them, optical flow refers to the movement of the target caused by the movement of the target, scene or camera between two consecutive frames of images; it is the two-dimensional vector field of the image during the translation process, and is a velocity field that represents the three-dimensional movement of the object point through a two-dimensional image, reflecting the image changes caused by the movement in a small time interval to determine the movement direction and movement rate of the image point. In this embodiment, the detection method of the optical flow feature can be implemented by an algorithm, and the specific implementation algorithm can adopt any algorithm known to those skilled in the art, such as but not limited to dense optical flow in optical flow analysis (an image registration method for point-by-point matching of images), sparse optical flow (a method for image registration specifically for sparse points on an image), etc., and for example, it can be extracted based on the optical flow extraction application program interface (Application Programming Interface, API) in openCV (a cross-platform computer vision and machine learning software library), and the disclosed embodiment is not limited to this.

[0064] For video files, optical flow reflects the changes in adjacent sequence pixels, while removing meaningless static content to prevent interference caused by huge changes in static content, which can make the model robust to different apps and different recognition scenarios. The optical flow extraction method in this embodiment includes but is not limited to dense optical flow, sparse optical flow or variants that can reflect pixel changes in optical flow analysis, etc. Figure 2In the present invention, the optical flow obtained based on the optical flow extraction application programming interface (API) in openCV is taken as an example to illustrate the principle of optical flow feature extraction.

[0065] For example, Figure 2 211 and 212 represent the content of the two frames before and after clicking the search button in the screen recording video; 213 represents the extracted optical flow feature map. Figure 2 It can be seen that the optical flow feature retains the change relationship of pixels while removing unnecessary static content.

[0066] Based on this, if an external camera is used to record video files, even if irrelevant content outside the screen is captured, as long as the picture of the irrelevant content remains relatively still in the two frames before and after, no information of the irrelevant content will be retained after optical flow extraction. Therefore, the extraction of optical flow features enables the method provided in this embodiment to avoid the disadvantage of the difficulty of data processing caused by the recording of irrelevant content when shooting with an external camera, while retaining the change characteristics of the APP screen recording, so that the recognition of key frames is more accurate and faster.

[0067] S103. Detect the optical flow features of the video file through a pre-trained key frame detection model to obtain the relative position of the start frame and the relative position of the end frame in the video file that responds to the detection operation; wherein the key frame detection model is a deep convolutional network, and takes the optical flow features of the video file to be tested as input, and takes the relative position of the start frame and the relative position of the end frame as output.

[0068] In this embodiment, the optical flow features of the video file are detected by a pre-trained key frame detection model, and the relative position of the start frame and the relative position of the end frame in the video file that responds to the detection operation can be determined. This step can determine the relative position of the start frame and the end frame in the entire video file, and provide data support for calculating the APP response time in subsequent steps. In some embodiments, the start frame position and the end frame position can be determined based on the relative position of the start frame and the relative position of the end frame and the frame rate of the video file.

[0069] In some embodiments, the entire video file is used as a reference, and the start frame relative position and the end frame relative position can both be positions relative to the start position (or other set positions) of the video file.

[0070] For example, Figure 3This is a schematic diagram of the key frame determination method in the method of the embodiment of the present disclosure, showing the key frames when the APP is cold started. Among them, when the APP is cold started, its response time (or startup time) is the time from clicking the APP icon to the completion of loading the text content on the homepage. Figure 3 , 221-225 respectively represent multiple frames of the APP startup process, in order from left to right and from top to bottom, taking clicking the APP icon in the 3rd row and 4th column as an example, the key frames include: the frame when clicking the APP, that is, the video frame represented by 222, which represents the starting frame of the response detection operation; and the frame after loading is completed, that is, the video frame represented by 224, which represents the ending frame of the response detection operation.

[0071] S104: Determine the response time of the application program according to the relative position of the start frame, the relative position of the end frame, and the frame rate of the video file.

[0072] In this embodiment, the starting frame position and the ending frame position can be determined based on the relative position of the starting frame and the relative position of the ending frame and the frame rate of the video file. The starting frame position and the ending frame position correspond to the duration occupied in the entire video file, which is the time taken for the application to respond to the detection operation.

[0073] Exemplarily, the method may be used to detect the time taken to start an APP, or the response time taken to perform other operations in the APP, which is not limited in the embodiments of the present disclosure.

[0074] In the application performance testing method provided by the embodiment of the present disclosure, the optical flow features of the video file are detected through the key frame detection model to obtain the relative position of the start frame and the relative position of the end frame in the video file that responds to the detection operation, and then the response time of the application is determined, thereby shielding the influence of meaningless interface changes and irrelevant pictures, and having strong adaptability to changes in the interface content of the application, so that the method has strong applicability; at the same time, the key frames are determined based on the optical flow features, rather than based on specific markers or stage classifications, which improves the efficiency and accuracy of determining the key frames (i.e., the start frame and the end frame), thereby improving the problem of low detection efficiency; in addition, due to the strong applicability and high robustness, the probability of successful detection is increased, and the proportion of manual review is reduced, thereby reducing the manpower cost of manual review after detection, and further improving the detection efficiency.

[0075] In some embodiments, Figure 1 Based on the above, S104 may include S1041 to S1044:

[0076] S1041, determining the number of frames of the video file based on the duration and frame rate of the video file.

[0077] The frame rate of a video file is a measure of the number of displayed frames, and its unit is the number of displayed frames per second (FPS) or Hertz (Hz). The number of frames of a video file can be obtained by multiplying the duration of the video file by the frame rate; for example, if the duration is A1 seconds (s) and the frame rate is A2 FPS, the number of frames is A1×A2.

[0078] S1042, determining the start frame position based on the start frame relative position and the number of frames in the video file; and determining the end frame position based on the end frame relative position and the number of frames in the video file.

[0079] The start frame relative position and the end frame relative position are used to represent the relative positions of the start frame and the end frame in the video file, respectively, and may be normalized positions. Thus, the start frame position may be determined by multiplying the start frame relative position by the number of frames in the video file; similarly, the end frame position may be determined by multiplying the end frame relative position by the number of frames in the video file.

[0080] In some embodiments, the starting frame position is y1×N; the ending frame position is y2×N; wherein y1 is the relative position of the starting frame, and y1∈[0,1]; y2 is the relative position of the ending frame, and y2∈[0,1]; and N is the number of frames in the video file.

[0081] Among them, normalizing the number of frames of the video file can simplify the data processing process of the key frame detection model. The relative position of the start frame and the relative position of the end frame are values ​​between 0 and 1 corresponding to the video file after the normalization process.

[0082] Exemplarily, in combination with the above N=A1×A2, and N is a positive integer; by multiplying the number of frames of the video file with the relative position of the start frame or the relative position of the end frame, the corresponding start frame position and end frame position can be obtained.

[0083] For example, y1=0.2, y2=0.8, N=500, then the start frame position is 100 frames, and the end frame position is 400 frames. This is only an exemplary description and does not constitute a limitation on the method provided in the embodiment of the present disclosure.

[0084] In the above embodiment, the key frame detection model is a deep convolution model, which can also be called a deep learning model. The deep convolution model can accurately and quickly determine the relative position of the start frame and the relative position of the end frame.

[0085] In some embodiments, the key frame detection model is a deep convolutional model with cross-layer connections, wherein the cross-layer connections are used to connect and splice the output results of convolutional layers of different depths to fuse the convolutional layer features of different time and space scales. It can be seen that the CNN feature extraction results of different time and space scales can be fused together through cross-layer connections, so that the network can adapt to and process videos of different time lengths, that is, the network can be applied to process video files of any different number of frames; at the same time, it can be applied to process video images with different width and height sizes of video screens.

[0086] In some embodiments, the deep convolution model with cross-layer connections includes: multiple convolution layers, multiple maximum downsampling layers, multiple adaptive downsampling layers, 1 splicing layer, and multiple fully connected layers; wherein the number of the multiple convolution layers is 1 more than the number of the multiple maximum downsampling layers;

[0087] The output of one of the multiple convolutional layers is connected to an adaptive downsampling layer; the output of each of the remaining convolutional layers is connected to a maximum downsampling layer, and the output of each maximum downsampling layer is connected to a convolutional layer of the next depth; and the depth of the convolutional layer connected to the adaptive downsampling layer is greater than the depth of each convolutional layer connected to the maximum downsampling layer;

[0088] The output of each convolutional layer connected to the maximum downsampling layer is cross-connected with an adaptive downsampling layer;

[0089] The outputs of multiple adaptive downsampling layers are connected to the concatenation layer;

[0090] The output of the concatenation layer is connected to a fully connected layer, and multiple fully connected layers are connected in series, and the output of the last fully connected layer connected in series is the relative position of the start frame and the relative position of the end frame of the response detection operation in the video file.

[0091] In some embodiments, Figure 4 Schematic diagram of a deep convolutional model in the method of an embodiment of the present disclosure, showing a deep convolutional model with cross-layer connections (hereinafter referred to as a "network" or "model"). Figure 4 The deep convolution model includes: 4 convolutional layers, 3 maximum downsampling (Maxpooling) layers, 4 adaptive downsampling (AdaptivePooling) layers, 1 concatenation (concatenate) layer and 2 fully connected (Dense Layer) layers.

[0092] The first convolution layer is used to perform convolution processing on the optical flow features of the video file. The output of the first convolution layer is connected to the first maximum downsampling layer, and the output of the first convolution layer is cross-layer connected to the first adaptive downsampling layer; the output of the first maximum downsampling layer is connected to the second convolution layer;

[0093] The output of the second convolutional layer is connected to the second maximum downsampling layer, and the output of the second convolutional layer is cross-layer connected to the second adaptive downsampling layer; the output of the second maximum downsampling layer is connected to the third convolutional layer;

[0094] The output of the third convolutional layer is connected to the third maximum downsampling layer, and the output of the third convolutional layer is cross-layer connected to the third adaptive downsampling layer; the output of the third maximum downsampling layer is connected to the fourth convolutional layer;

[0095] The output of the fourth convolutional layer is connected to the fourth adaptive downsampling layer;

[0096] The output of the first adaptive downsampling layer, the output of the second adaptive downsampling layer, the output of the third adaptive downsampling layer and the output of the fourth adaptive downsampling layer are all connected to the splicing layer;

[0097] The output of the concatenated layer is connected to the first fully connected layer; the output of the first fully connected layer is connected to the second fully connected layer;

[0098] The output of the second fully connected layer is the relative position of the start frame and the relative position of the end frame of the response detection operation in the video file.

[0099] It can be seen that the order in which the optical flow features are processed by each layer is: the first convolution layer-the first maximum downsampling layer-the second convolution layer-the second maximum downsampling layer-the third convolution layer-the third maximum downsampling layer-the fourth convolution layer; the first adaptive downsampling layer in the four adaptive downsampling layers samples the data processed by the first convolution layer, the second adaptive downsampling layer samples the data processed by the second convolution layer, the third adaptive downsampling layer samples the data processed by the third convolution layer, and the fourth adaptive downsampling layer samples the data processed by the fourth convolution layer; the data obtained by the four adaptive downsampling layers are processed by the splicing layer and then input into the first fully connected layer-the second fully connected layer in sequence, and the output of the second fully connected layer is the relative position of the start frame and the relative position of the end frame in the video file that responds to the detection operation.

[0100] Exemplarily, the graph convolution layer may be a 3D-CNN convolution layer. In this embodiment, a nonlinear activation function may also be used after the convolution, which is not shown in the figure. The 3D-CNN architecture can generate multiple information channels from adjacent optical flow feature images, and perform convolution and downsampling in each channel respectively, and obtain the final feature representation by combining the information from the optical flow feature channel. The adaptive downsampling layer can accept a tensor input of any dimension, evenly divide the input of the current dimension into grids of a specified dimension, downsample each small grid separately, and finally output a tensor of a specified dimension, so that the network can accept inputs of any frame length and any width and height. The fully connected layer can splice the tensors output by the adaptive downsampling layers corresponding to the four convolutional layers according to the channel dimension.

[0101] Figure 4 In the figure, [1], [2], [3] and [4] are cross-layer connection structures, which can connect and concatenate the output results of 3D-CNN convolutional layers of different depths. Among them, [1] is the output after processing by the first convolutional layer, [2] is the output after processing by the second convolutional layer, and the others are similar and will not be repeated here. Through cross-layer connection, the CNN feature extraction results of different time and space scales can be fused together, so that the network can adapt to and process videos of different time lengths, that is, the network can be applied to process video files of any different number of frames; at the same time, it can be applied to process video images with different width and height sizes.

[0102] After the cross-layer connection, all outputs are merged through the concatenation layer to obtain a vector of fixed dimension. The vector is passed through two fully connected layers, and finally output y1 and y2, which are the relative positions of the start frame and the end frame.

[0103] Among them, the output result is first normalized by the sigmoid function to standardize the output value to between 0 and 1, that is, y1∈[0,1] and y2∈[0,1].

[0104] In other implementations, parameters such as the specific structure, number of layers, and type of nonlinear activation function applied of the deep learning model can be flexibly adjusted and set according to the processing requirements of the specific video file, and the embodiments of the present disclosure are not limited to this.

[0105] In other embodiments, other training methods such as data enhancement and regularization may be used in conjunction with the model training process to enhance the training accuracy of the model.

[0106] In some embodiments, based on the start frame position and the end frame position, determining the response time of the application to the detection operation may include the following steps 1 to 3:

[0107] Step 1: Based on the frame rate of the video file, determine the duration between two adjacent frames.

[0108] Among them, if the frame rate is A2FPS, the duration between two adjacent frames is 1 / A2.

[0109] For example, if the value of A2 is 10, the duration between two adjacent frames is 0.1 s. This is only an exemplary description and does not constitute a limitation on the method provided in the embodiment of the present disclosure.

[0110] Step 2: Based on the start frame position and the end frame position, determine the number of frames included between the start frame and the end frame.

[0111] For the response process of the detection operation, the number of frames between the start frame and the end frame is the number of response frames corresponding to the detection operation.

[0112] Exemplarily, the number of frames corresponding to the response time may be equal to the number of frames included between the start frame and the end frame; combined with the above, it can be expressed as: y2×N-y1×N.

[0113] Step 3: Based on the duration between two adjacent frames and the number of frames between the start frame and the end frame, determine the time it takes for the application to respond to the operation.

[0114] The duration between two adjacent frames is the single frame duration. The response time can be obtained by multiplying the single frame duration by the number of frames corresponding to the response time. Combined with the above, the response time can be expressed as: (y2×N-y1×N) / A2.

[0115] In some embodiments, the frame rate of the video file is a fixed frame rate.

[0116] In this way, the duration of a single frame is fixed, and the calculation method of the response time is relatively simple.

[0117] In other implementations, when the frame rate of the video file is a non-fixed frame rate, the response time may be determined by summing the duration of each frame between the start frame and the end frame, which is not limited in this embodiment of the present disclosure.

[0118] Based on the above implementation, Figure 5 A schematic diagram comparing the method provided in the embodiment of the present disclosure with the related technology shows a comparison of manual labeling, frame-by-frame calculation and classification, and labeling, review, total time consumption, and manpower time consumption (i.e., manpower cost) applied to APP cold start based on optical flow features and deep convolutional models.

[0119] Among them, in the actual test of APP response time performance, the test was carried out on three types of models, 10 low, medium and high, and each indicator of each model was tested 50 times, and the length of each video file was about 10 seconds. To complete the test of a response time indicator, a total of 1,500 screen recording videos (i.e. video files) of APP operations need to be tested. In the case of automated screen recording, 90% of the screen recording videos are started by adb commands, and about 80% of the videos cannot see the starting stage, and thus cannot be located to the starting frame position by the traditional frame-by-frame calculation and classification method.

[0120] The annotation cost of a single screen recording video is as follows:

[0121] In the manual labeling method: labeling takes 60 seconds, and review takes 20 seconds; and the review rate of manual labeling is about 30%. In frame-by-frame calculation and classification: the algorithm takes 65 seconds, and the review takes 60 seconds. If the video is started by the adb command and the beginning stage cannot be seen, the detection will fail. And the failure rate of detection is about 90%*80%=72%. In the deep learning model based on optical flow: the algorithm takes 10 seconds, and the review takes 60 seconds. About 5% of the videos need to be reviewed due to large errors, but the detection will not fail due to the missing beginning stage.

[0122] The time cost of testing single response time-consuming indicators is shown in Table 1.

[0123] Table 1 Time comparison table

[0124] Algorithm\Cost(hours) Annotation Review Total time Manpower time-consuming Manual annotation 25.00 8.30 33.33 33.33 Calculate and classify frame by frame 27.08 18.00 45.08 18.00 Deep learning model based on optical flow 4.17 1.25 5.42 1.25

[0125] From the above performance comparison, we can see that the traditional frame-by-frame calculation and classification algorithm still requires high manual review costs due to its high failure rate. At the same time, the algorithm itself is time-consuming.

[0126] The deep learning model based on optical flow features has greater applicability and a lower detection failure rate, so the final manual review cost is low, and the manpower time consumed is only 4% of manual labeling.

[0127] Compared with traditional frame-by-frame calculation and classification algorithms, the deep learning model based on optical flow is faster, more applicable, and can save manpower testing costs and testing time costs to a large extent.

[0128] Among them, the reason why the detection efficiency of methods based on markers or frame-by-frame calculation of objective indicators is low is: First, the video recording tools are diversified. Specifically, built-in screen recording tools can be used: for example, the Android system can record the screen by built-in adb command in the APP, but there is an unstable frame rate. In addition, enabling the screen recording function in the machine under test will occupy some system resources, especially on machines with lower performance, which will affect the response time of the program under test and make the performance index extraction inaccurate. External camera recording can also be used: for example, using a camera to shoot a mobile phone, it is inevitable that content outside the screen will be shot, such as the mobile phone shell, desktop, etc. In addition, the position, angle, and even ambient light of the object being photographed will affect the content of the video shooting. This makes the subsequent frame automation calculation error large or even invalid. Second, the diversity of content in the same scene of the APP. Specifically, the frequent changes of APP filling content and personalized configuration such as night mode will affect the performance or parameter configuration of the traditional frame algorithm. Third, frame-by-frame calculation is a waste of computing power. Specifically, the detection based on markers has a decrease in detection efficiency as the markers change or increase in number.

[0129] The application performance testing method provided by the embodiment of the present disclosure detects the optical flow features of the video file through a key frame detection model to obtain the relative position of the start frame and the relative position of the end frame in the video file that responds to the detection operation, and then determines the response time of the application. It can shield the impact of meaningless interface changes and irrelevant pictures, and has a strong adaptability to changes in APP interface content and stronger compatibility. At the same time, the key frame detection model is adopted, which has a higher computational efficiency. Compared with the general method based on frame-by-frame calculation of objective indicators such as PSNR, the method can realize the determination of key frames based on a deep learning model, thereby determining the key frames faster, which is conducive to improving detection efficiency.

[0130] On the basis of the above implementation, the embodiment of the present disclosure further provides an application performance testing device, which can execute any of the above methods. Therefore, the device also has the beneficial effects of any of the methods in the above implementation, and the similarities can be understood by referring to the explanation of the method above, which will not be repeated below.

[0131] For example, Figure 6 A schematic diagram of the structure of an application performance testing device provided by an embodiment of the present disclosure. Figure 6The device may include: an acquisition unit 301, used to acquire a video file generated for executing a test action for an application; the video file can show the interface changes of the application in response to the test operation; an extraction unit 302, used to extract features from the video file to obtain optical flow features of the video file; a first determination unit 303, used to detect the optical flow features of the video file through a pre-trained key frame detection model to obtain the relative position of the start frame and the relative position of the end frame in the video file that responds to the detection operation; wherein the key frame detection model is a deep convolutional network, and takes the optical flow features of the video file to be tested as input, and takes the relative position of the start frame and the relative position of the end frame as output; a second determination unit 304, used to determine the response time of the application according to the relative position of the start frame and the relative position of the end frame and the frame rate of the video file.

[0132] In the application performance testing device provided by the embodiment of the present disclosure, the relative position of the start frame and the relative position of the end frame are determined based on the optical flow features instead of directly extracting features based on the RGB image, thereby shielding the influence of meaningless interface changes and irrelevant pictures, and having strong adaptability to changes in the interface content of the application, so that the method has strong applicability; at the same time, the relative position of the start frame and the relative position of the end frame are determined based on the optical flow features instead of based on specific markers or stage classifications, which improves the efficiency and accuracy of determining the key frames (i.e., the start frame and the end frame), thereby improving the problem of low detection efficiency; in addition, due to the strong applicability and high robustness, the probability of successful detection is increased, and the proportion of manual review is reduced, thereby reducing the manpower cost of manual review after detection, and further improving the detection efficiency.

[0133] In some embodiments, the key frame detection model is a deep convolutional model with cross-layer connections; wherein the cross-layer connections are used to connect and splice the results of the outputs of convolutional layers of different depths to fuse the convolutional layer features of different time and space scales.

[0134] In this way, the key frame detection model can be used to take the optical flow features as input to obtain the relative position of the start frame and the relative position of the end frame in the video file that responds to the detection operation, which is highly efficient and accurate.

[0135] In some embodiments, the deep convolution model with cross-layer connections includes: multiple convolutional layers, multiple maximum downsampling layers, multiple adaptive downsampling layers, 1 splicing layer, and multiple fully connected layers; wherein the number of the multiple convolutional layers is 1 more than the number of the multiple maximum downsampling layers;

[0136] The output of one of the multiple convolutional layers is connected to an adaptive downsampling layer; the output of each of the remaining convolutional layers is connected to a maximum downsampling layer, and the output of each maximum downsampling layer is connected to a convolutional layer of the next depth; and the depth of the convolutional layer connected to the adaptive downsampling layer is greater than the depth of each convolutional layer connected to the maximum downsampling layer;

[0137] The output of each convolutional layer connected to the maximum downsampling layer is cross-connected with an adaptive downsampling layer;

[0138] The outputs of multiple adaptive downsampling layers are connected to the concatenation layer;

[0139] The output of the concatenation layer is connected to a fully connected layer, and multiple fully connected layers are connected in series, and the output of the last fully connected layer connected in series is the relative position of the start frame and the relative position of the end frame of the response detection operation in the video file.

[0140] In some embodiments, the deep convolutional model with cross-layer connections includes: 4 convolutional layers, 3 maximum downsampling layers, 4 adaptive downsampling layers, 1 splicing layer and 2 fully connected layers;

[0141] The first convolution layer is used to perform convolution processing on the optical flow features of the video file. The output of the first convolution layer is connected to the first maximum downsampling layer, and the output of the first convolution layer is cross-layer connected to the first adaptive downsampling layer; the output of the first maximum downsampling layer is connected to the second convolution layer;

[0142] The output of the second convolutional layer is connected to the second maximum downsampling layer, and the output of the second convolutional layer is cross-layer connected to the second adaptive downsampling layer; the output of the second maximum downsampling layer is connected to the third convolutional layer;

[0143] The output of the third convolutional layer is connected to the third maximum downsampling layer, and the output of the third convolutional layer is cross-layer connected to the third adaptive downsampling layer; the output of the third maximum downsampling layer is connected to the fourth convolutional layer;

[0144] The output of the fourth convolutional layer is connected to the fourth adaptive downsampling layer;

[0145] The output of the first adaptive downsampling layer, the output of the second adaptive downsampling layer, the output of the third adaptive downsampling layer and the output of the fourth adaptive downsampling layer are all connected to the splicing layer;

[0146] The output of the concatenated layer is connected to the first fully connected layer; the output of the first fully connected layer is connected to the second fully connected layer;

[0147] The output of the second fully connected layer is the relative position of the start frame and the relative position of the end frame of the response detection operation in the video file.

[0148] It can be seen that the order in which the optical flow features are processed by each layer is: the first convolution layer-the first maximum downsampling layer-the second convolution layer-the second maximum downsampling layer-the third convolution layer-the third maximum downsampling layer-the fourth convolution layer; the first adaptive downsampling layer in the four adaptive downsampling layers samples the data processed by the first convolution layer, the second adaptive downsampling layer samples the data processed by the second convolution layer, the third adaptive downsampling layer samples the data processed by the third convolution layer, and the fourth adaptive downsampling layer samples the data processed by the fourth convolution layer; the data obtained by the four adaptive downsampling layers are processed by the splicing layer and then input into the first fully connected layer-the second fully connected layer in sequence, and the output of the second fully connected layer is the relative position of the start frame and the relative position of the end frame in the video file that responds to the detection operation.

[0149] In this way, the deep convolution model can be used to determine the relative position of the start frame and the relative position of the end frame. The network is relatively simple and can be applied to video files with any different frame numbers and width and height sizes.

[0150] In some embodiments, the second determining unit 304 determines the response time of the application according to the relative position of the start frame and the relative position of the end frame and the frame rate of the video file, including:

[0151] Determine the number of frames of the video file based on the duration and frame rate of the video file;

[0152] Determine the start frame position based on the relative position of the start frame and the number of frames in the video file;

[0153] Determine the end frame position based on the relative position of the end frame and the number of frames in the video file;

[0154] Based on the start frame position and the end frame position, determine how long it takes the application to respond to the detection operation.

[0155] In this way, the start frame position and the end frame position may be determined based on the duration and frame rate of the video file and in combination with the start frame relative position and the end frame relative position.

[0156] In some embodiments, the starting frame position is y1×N; the ending frame position is y2×N; wherein y1 is the relative position of the starting frame, and y1∈[0,1]; y2 is the relative position of the ending frame, and y2∈[0,1]; and N is the number of frames in the video file.

[0157] In this way, the length of the video file can be normalized, and based on this, the relative position of the starting frame and the relative position of the ending frame can be determined, and the starting frame position and the ending frame position can be further determined, thereby simplifying the deep convolution model, which is beneficial to improving the data processing rate, thereby improving the detection rate.

[0158] In some embodiments, the second determining unit 304 determines the time taken for the application to respond to the detection operation based on the start frame position and the end frame position, including:

[0159] Based on the frame rate of the video file, determine the duration between two adjacent frames;

[0160] Based on the start frame position and the end frame position, determining the number of frames included between the start frame and the end frame;

[0161] The time taken for the application to respond to the detection operation is determined based on the duration between two adjacent frames and the number of frames included between the start frame and the end frame.

[0162] In this way, the time it takes for the application to respond to the detection operation can be determined based on the frame rate of the video file and in combination with the start frame position and the end frame position.

[0163] In some embodiments, the frame rate of the video file is a fixed frame rate.

[0164] In this way, the data processing process can be simplified, which is beneficial to improving detection efficiency.

[0165] The present disclosure also provides an electronic device, which includes: a processor and a memory; the processor is used to execute the steps of any of the above methods by calling the program or instruction stored in the memory. Therefore, the electronic device also has the beneficial effects of the above methods and devices, and the similarities can be understood by referring to the explanation of the method and device above, and will not be repeated below.

[0166] In some embodiments, Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 7 , the electronic device comprises:

[0167] One or more processors 401, Figure 7 A processor 401 is taken as an example;

[0168] Memory 402;

[0169] The electronic device may further include: an input device 403 and an output device 404 .

[0170] The processor 401, memory 402, input device 403 and output device 404 in the electronic device can be connected via a bus or other means. Figure 7 The connection method is illustrated by taking bus connection as an example.

[0171] The memory 402 is a non-transitory computer-readable storage medium that can be used to store software programs, computer executable programs, and modules, such as program instructions / modules / units corresponding to any of the above methods of the application program in the embodiments of the present disclosure (for example, the attached Figure 6 The processor 401 executes various functional applications and data processing of the server by running the software programs, instructions, units and modules stored in the memory 402, that is, the method of implementing the above method embodiment.

[0172] The memory 402 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created according to the use of the electronic device, etc.

[0173] In addition, the memory 402 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0174] In some embodiments, the memory 402 may optionally include a memory remotely arranged relative to the processor 401, and these remote memories may be connected to the terminal device via a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0175] The input device 403 may be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the electronic device.

[0176] The output device 404 may include a display device such as a display screen.

[0177] The present disclosure also provides a non-transitory computer-readable storage medium, which stores a program or instruction, and the program or instruction enables a computer to execute the steps of any of the above methods.

[0178] Through the above description of the implementation method, the technicians in the relevant field can clearly understand that the above method of the embodiment of the present disclosure can be implemented by software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the essence of the above method related technical solutions of the embodiment of the present disclosure or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer's floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the various methods of the embodiment of the present disclosure.

[0179] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0180] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for testing application performance, characterized in that: include: Obtaining a video file generated by executing a test action on the application; The video file can show the interface changes of the application program in response to the test operation; Extracting features from the video file to obtain optical flow features of the video file, so as to remove still picture content from the video file; The optical flow features of the video file are detected by a pre-trained key frame detection model to obtain the relative position of the start frame and the relative position of the end frame in the video file that responds to the detection operation; wherein the key frame detection model is a deep convolutional network, and takes the optical flow features of the video file to be tested as input, and takes the relative position of the start frame and the relative position of the end frame as output; The response time of the application is determined according to the relative position of the start frame, the relative position of the end frame and the frame rate of the video file.

2. The method according to claim 1, characterized in that The key frame detection model is a deep convolutional model with cross-layer connections, wherein the cross-layer connections are used to connect and splice the results of convolutional layer outputs of different depths to fuse convolutional layer features of different time and space scales.

3. The method according to claim 2, characterized in that The deep convolution model includes: a plurality of convolutional layers, a plurality of maximum downsampling layers, a plurality of adaptive downsampling layers, a splicing layer and a plurality of fully connected layers; wherein the number of the plurality of convolutional layers is one more than the number of the plurality of maximum downsampling layers; The output of one of the multiple convolutional layers is connected to an adaptive downsampling layer; the output of each of the remaining convolutional layers is connected to a maximum downsampling layer, and the output of each maximum downsampling layer is connected to a convolutional layer of the next depth; and the depth of the convolutional layer connected to the adaptive downsampling layer is greater than the depth of each convolutional layer connected to the maximum downsampling layer; The output of each convolutional layer connected to the maximum downsampling layer is cross-connected with an adaptive downsampling layer; The outputs of the multiple adaptive downsampling layers are all connected to the splicing layer; The output of the splicing layer is connected to a fully connected layer, and the multiple fully connected layers are connected in series, and the output of the last fully connected layer connected in series is the relative position of the start frame and the relative position of the end frame in the video file that responds to the detection operation.

4. The method according to claim 3, characterized in that The deep convolution model includes: 4 convolutional layers, 3 maximum downsampling layers, 4 adaptive downsampling layers, 1 splicing layer and 2 fully connected layers; The first convolution layer is used to perform convolution processing on the optical flow features of the video file, the output of the first convolution layer is connected to the first maximum downsampling layer, and the output of the first convolution layer is cross-layer connected to the first adaptive downsampling layer; the output of the first maximum downsampling layer is connected to the second convolution layer; The output of the second convolutional layer is connected to the second maximum downsampling layer, and the output of the second convolutional layer is cross-layer connected to the second adaptive downsampling layer; the output of the second maximum downsampling layer is connected to the third convolutional layer; The output of the third convolutional layer is connected to the third maximum downsampling layer, and the output of the third convolutional layer is cross-layer connected to the third adaptive downsampling layer; the output of the third maximum downsampling layer is connected to the fourth convolutional layer; The output of the fourth convolutional layer is connected to the fourth adaptive downsampling layer; The output of the first adaptive downsampling layer, the output of the second adaptive downsampling layer, the output of the third adaptive downsampling layer and the output of the fourth adaptive downsampling layer are all connected to the splicing layer; The output of the concatenated layer is connected to the first fully connected layer; the output of the first fully connected layer is connected to the second fully connected layer; The output of the second fully connected layer is the relative position of the start frame and the relative position of the end frame in the video file that responds to the detection operation.

5. The method according to claim 1, characterized in that Determining the response time of the application according to the relative position of the start frame, the relative position of the end frame and the frame rate of the video file includes: Determine the number of frames of the video file based on the duration and frame rate of the video file; Determine the starting frame position based on the starting frame relative position and the number of frames of the video file; Determining an end frame position based on the end frame relative position and the number of frames of the video file; Based on the start frame position and the end frame position, a time consumption of the application program in responding to the detection operation is determined.

6. The method according to claim 5, characterized in that The determining, based on the start frame position and the end frame position, the response time of the application program to the detection operation comprises: Determine the duration between two adjacent frames based on the frame rate of the video file; Based on the start frame position and the end frame position, determining the number of frames included between the start frame and the end frame; The time taken for the application to respond to the detection operation is determined based on the duration between two adjacent frames and the number of frames between the start frame and the end frame.

7. An application performance testing device, characterized in that: include: An acquisition unit, used for acquiring a video file generated by executing a test action on an application program; The video file can show the interface changes of the application program in response to the test operation; An extraction unit, configured to extract features from the video file to obtain optical flow features of the video file, so as to remove still images from the video file; A first determination unit is used to detect the optical flow features of the video file through a pre-trained key frame detection model to obtain the relative position of the start frame and the relative position of the end frame in the video file that responds to the detection operation; wherein the key frame detection model is a deep convolutional network, and takes the optical flow features of the video file to be tested as input, and takes the relative position of the start frame and the relative position of the end frame as output; The second determining unit is used to determine the response time of the application according to the relative position of the start frame, the relative position of the end frame and the frame rate of the video file.

8. The device according to claim 7, characterized in that The key frame detection model is a deep convolutional model with cross-layer connections, wherein the cross-layer connections are used to connect and splice the results of convolutional layer outputs of different depths to fuse convolutional layer features of different time and space scales.

9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory; the processor is used to execute the steps of the method according to any one of claims 1 to 6 by calling the program or instructions stored in the memory.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores a program or instruction, which enables a computer to execute the steps of the method according to any one of claims 1 to 6.

Images