Display interface fluency testing method and device and computer readable storage medium

Abstract

The invention discloses a display interface fluency testing method and testing equipment and a computer readable storage medium, relates to the technical field of automatic testing, and aims to solve the problem of how to accurately test the fluency of a display interface. The testing method comprises the steps that N frames of testing RGB images are obtained, the testing RGB images comprise a display interface of a terminal to be tested, the display interface comprises a target control area, and N is a positive integer and Ngt; 2. And converting the N frames of test RGB images located in the camera coordinate system into N frames of correction images located in the to-be-tested terminal coordinate system. And converting the N frames of corrected images into N frames of test HSV images. And performing color filtering on the N frames of test HSV images to obtain N frames of test mask images. Acquiring N test rectangles corresponding to the contours of the N target control areas in the N frames of test mask images, wherein the test rectangles are the minimum enclosing rectangles of the contours of the target control areas; and when the center point coordinates of the N test rectangles meet a fluency condition, determining that the display interface of the to-be-tested terminal is fluent.

Description

Technical Field

[0001] This application relates to the field of automated testing technology, specifically to a testing method, testing equipment, and computer-readable storage medium for testing the smoothness of a display interface. Background Technology

[0002] With the rapid proliferation of smartphones, tablets, and other mobile devices, the smoothness of the display interface has a significant impact on user experience. Currently, the smoothness of the display interface is mainly assessed by the frame rate (Frames Per Second, FPS) of the video stream when playing videos. The frame rate is the number of image frames displayed or captured per second. However, the frame rate of the video stream cannot fully reflect the smoothness of the display interface. Taking input latency as an example, input latency is the time from user input to screen response. When the video stream has a high frame rate and a large input latency, the frame rate can indicate a smooth display interface, but a large input latency will make the user noticeably sluggish, leading to a misjudgment of the smoothness of the display interface. Summary of the Invention

[0003] In view of this, embodiments of this application provide a method, testing device, and computer-readable storage medium for testing the smoothness of a display interface, aiming to solve the problem of how to accurately test the smoothness of a display interface.

[0004] The first aspect of this application provides a method for testing the smoothness of a display interface, applied to a testing device. The method includes: acquiring N test RGB images, each containing the display interface of a terminal under test, the display interface including target control areas, the color of which differs from the background color of the display interface. N is a positive integer, and N>2. The N test RGB images located in the camera coordinate system are converted into N corrected images located in the coordinate system of the terminal under test. The N corrected images are converted into N test HSV images. The N test HSV images are color filtered to obtain N test mask images. N test rectangles are acquired corresponding to the contours of N target control areas in the N test mask images, each test rectangle being the minimum bounding rectangle of the contour of the target control area. When the center point coordinates of the N test rectangles satisfy the smoothness condition, the display interface of the terminal under test is determined to be smooth.

[0005] In this embodiment, N test rectangles corresponding to the outlines of N target control areas are obtained to determine whether the center point coordinates of the N test rectangles meet the smoothness condition. The center point coordinates of the N test rectangles reflect the dynamic effect of the target control on the display interface. When the center point coordinates of the N test rectangles meet the smoothness condition, it indicates that the change process of the dynamic effect of the target control on the display interface is smooth, thus determining that the display interface of the terminal under test is smooth. The change process of the dynamic effect of the target control is a user-perceptible indicator. Using a user-perceptible indicator to evaluate the smoothness of the terminal's display interface can accurately reflect the user's intuitive feeling about the smoothness of the display interface, thereby improving the accuracy of the test. Moreover, this embodiment can effectively detect various dynamic effects of the target control, such as sliding icons or windows, opening or closing applications, etc., and has a wide range of applications.

[0006] In one embodiment, the method further includes: performing grayscale transformation on N frames of corrected images to obtain N frames of grayscale images. Traversing the N frames of grayscale images, the grayscale difference between the region of interest in the (i+1)th frame and the region of interest in the i-th frame is calculated. After traversal, N-1 grayscale difference values ​​are obtained. i is a positive integer, and 1 ≤ i ≤ N-1. The start frame and end frame of the dynamic effect are determined based on the N-1 grayscale difference values. The duration of the dynamic effect is calculated based on the frame number of the start frame, the frame number of the end frame, and the frame rate of the test video. When the center point coordinates of the N test rectangles meet the smoothness condition, and the duration of the dynamic effect is less than or equal to the duration threshold, the display interface of the terminal under test is determined to be smooth.

[0007] In this embodiment, the duration of dynamic effects is also a user-perceptible indicator. Using the duration of dynamic effects as another indicator for evaluating the smoothness of the display interface can more accurately reflect the user's intuitive perception of the smoothness of the display interface, thereby improving the accuracy of the test. Furthermore, since contour extraction algorithms are easily affected by environmental noise, environmental color, and ambient lighting, the contour extraction error may be large, leading to low contour extraction accuracy. Therefore, by converting grayscale and using the frame difference method to calculate the grayscale difference between two consecutive grayscale images, and calculating the duration of the dynamic effect based on the grayscale difference, the interference of contour extraction error on the calculation of the dynamic effect duration can be effectively overcome, thereby accurately identifying the start and end frames of the dynamic effect and improving the accuracy of the test.

[0008] In another embodiment, determining the start and end frames of the dynamic effect based on N-1 grayscale differences includes: traversing the N-1 grayscale differences; when the i-th grayscale difference is greater than the grayscale threshold of the start frame, the i-th grayscale image is determined as the target start frame. When the i-th grayscale difference is less than the grayscale threshold of the end frame, the i-th grayscale image is determined as the target end frame. After traversal, K target start frames and T target end frames are obtained. The i-th grayscale difference is the grayscale difference between the region of interest in the (i+1)-th grayscale image and the region of interest in the i-th grayscale image. K and T are both positive integers, and 1 ≤ K ≤ N-1, 1 ≤ T ≤ N-1. The grayscale image with the smallest frame count among the K target start frames is determined as the start frame of the dynamic effect. The grayscale image with the smallest frame count among the T target end frames is determined as the end frame of the dynamic effect.

[0009] In another embodiment, the method further includes: when the center point coordinates, height and width of the N test rectangles all meet the smoothness condition, it is determined that the display interface of the terminal under test is smooth.

[0010] In this embodiment, the smoothness of the display interface is evaluated by combining the center point coordinates, height, and width of N test rectangles. This can more comprehensively reflect the dynamic effects of the target controls on the display interface, thereby more accurately reflecting the change process of the dynamic effects of the target controls on the display interface and improving the accuracy of the test.

[0011] In another embodiment, the method further includes: when the center point coordinates, height and width of the N test rectangles all meet the smoothness condition, and the duration of the dynamic effect is less than or equal to the duration threshold, the display interface of the terminal under test is determined to be smooth.

[0012] In another embodiment, the smoothness conditions for the center point coordinates, height, and width of the N test rectangles include: the dynamic effect curves corresponding to the center point coordinates, height, and width of the N test rectangles all satisfy the following: the variance of all point values ​​is less than or equal to the variance threshold, or the standard deviation of all point values ​​is less than or equal to the standard deviation threshold, or the average absolute difference between each adjacent point value is less than or equal to the average absolute difference threshold, or the average squared difference between each adjacent point value is less than or equal to the average squared difference threshold, or the deviation between the dynamic effect curve and the fitted curve is within the deviation range.

[0013] In another embodiment, the smoothness conditions for the center point coordinates, height, and width of the N test rectangles include: the arrays corresponding to the center point coordinates, height, and width of the N test rectangles all satisfy the following: the variance of all values ​​is less than or equal to the variance threshold, or the standard deviation of all values ​​is less than or equal to the standard deviation threshold, or the mean absolute difference between each adjacent value is less than or equal to the mean absolute difference threshold, or the mean squared difference between each adjacent value is less than or equal to the mean squared difference threshold.

[0014] In another embodiment, color filtering of N frames of test HSV images to obtain N frames of test mask images includes: color filtering of the N frames of test HSV images according to the j-th test target color filtering range, to obtain N frames of j-th test mask images, where j is a positive integer. The average value and standard deviation of the hue, saturation, and brightness of the j-th test target color region in the N frames of j-th test mask images are calculated. The (j+1)-th test target color filtering range is set according to the adjustment parameters and the average value and standard deviation of the hue, saturation, and brightness of the j-th test target color region. The (j+1)-th test target color filtering range is smaller than the j-th test target color filtering range; the adjustment parameters include hue adjustment parameters, saturation adjustment parameters, and brightness adjustment parameters. Color filtering of the N frames of test HSV images according to the (j+1)-th test target color filtering range is then performed to obtain N frames of (j+1)-th test mask images.

[0015] In this embodiment, the hue, saturation, and brightness values ​​of the test HSV image are significantly affected by ambient lighting and color. The j-th test target color has a wide filtering range. By setting the adjustment parameters corresponding to hue, saturation, and brightness to set the filtering range of the (j+1)-th test target color, the accuracy of color filtering can be improved. Furthermore, the adjustment parameters corresponding to hue, saturation, and brightness can be flexibly set, thereby enhancing the flexibility of color filtering. Because the color filtering range is reduced, the size of the test target color region is reduced, thus reducing the influence of ambient lighting and color on the test target color region in the test HSV image, thereby enhancing the robustness of the test target color region under different ambient lighting and color conditions.

[0016] In another embodiment, before converting the N-frame test RGB images located in the camera coordinate system into N-frame calibration images located in the coordinate system of the terminal under test, the method further includes: acquiring a calibration RGB image, the calibration RGB image containing the display interface of the terminal under test, the display interface displaying the calibration image, the calibration image including M preset regions, where M is a positive integer. The colors of the preset regions are different from the background color of the calibration image. Converting the calibration RGB image into a calibration HSV image. Performing color filtering on the calibration HSV image to obtain a calibration mask image. Acquiring M calibration rectangles corresponding to the contours of the M calibration target color regions in the calibration mask image, the calibration rectangles being the smallest bounding rectangles of the contours of the calibration target color regions. Calculating a perspective transformation matrix based on the center point coordinates of the M calibration rectangles and the center point coordinates of the M preset regions in the calibration image, the perspective transformation matrix being used to convert the N-frame test RGB images located in the camera coordinate system into N-frame calibration images located in the coordinate system of the terminal under test.

[0017] In another embodiment, obtaining N test rectangles corresponding to the contours of N target control regions in N frame test mask images includes: performing morphological processing on the N frame test mask images to obtain N frame morphological images; extracting the contours of N target control regions from the N frame morphological images; and obtaining N test rectangles corresponding to the contours of the N target control regions.

[0018] In another embodiment, obtaining M calibration rectangles corresponding to the contours of M calibration target color regions in the calibration mask image includes: extracting the contours of the M calibration target color regions from the calibration mask image; and obtaining the M calibration rectangles corresponding to the contours of the M calibration target color regions.

[0019] A second aspect of this application provides a testing device, which includes a camera and a control circuit, with the camera and control circuit communicatively connected. The camera, under the control of the control circuit, captures images of the display interface of the terminal under test within its field of view, acquires the captured images, and sends the captured images to the control circuit. The captured images include N frames of test RGB images, each containing the display interface of the terminal under test. The display interface includes target control areas, the colors of which are different from the background color of the display interface. N is a positive integer, and N>2. The control circuit acquires the N frames of test RGB images, converts the N frames of test RGB images located in the camera coordinate system into N frames of corrected images located in the coordinate system of the terminal under test, and converts the N frames of corrected images into N frames of test HSV images. Color filtering is performed on the N frames of test HSV images to obtain N frames of test mask images. N test rectangles corresponding to the contours of the N target control areas in the N frames of test mask images are acquired, where each test rectangle is the smallest bounding rectangle of the contour of the target control area. When the center point coordinates of the N test rectangles satisfy a smoothness condition, the display interface of the terminal under test is determined to be smooth.

[0020] In one embodiment, the testing equipment further includes a robotic arm and a drive circuit. The robotic arm includes an end effector for mounting a stylus, which performs touch operations on the touchscreen of the terminal under test. The drive circuit is electrically connected to the control circuit and the robotic arm. The drive circuit generates drive signals under the control of the control circuit and sends drive signals to the robotic arm. The drive signals drive the end effector of the robotic arm to move along a target trajectory, thereby causing the stylus to perform touch operations on the touchscreen of the terminal under test.

[0021] In another embodiment, the testing equipment further includes a fixed bracket with a guide rail mounted on it, and a slider mounted on the camera. The slider is used to mount onto the guide rail. When the slider moves along the length of the guide rail, it moves the camera so that the camera's field of view can cover the display interface of the terminal under test. The guide rail is horizontal in length.

[0022] A third aspect of this application provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, implements the method for testing the smoothness of the display interface provided in the first aspect.

[0023] The fourth aspect of this application provides a computer program product, which includes computer instructions, and implements the method for testing the smoothness of the display interface provided in the first aspect when the processor executes the computer instructions.

[0024] It is understood that the beneficial effects of the test equipment provided in the second aspect of the embodiments of this application, the computer-readable storage medium provided in the third aspect, and the computer program product provided in the fourth aspect are substantially the same as the beneficial effects of the test method for the smoothness of the display interface provided in the first aspect, and will not be repeated here. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of a test scenario provided as an example.

[0026] Figure 2 This is a schematic diagram of another example of a test scenario.

[0027] Figure 3 This is a schematic diagram of another example of a test scenario.

[0028] Figure 4 This is a flowchart of a test method provided as an example.

[0029] Figure 5 This is the first image provided as an example.

[0030] Figure 6 This is a calibration image provided as an example.

[0031] Figure 7 This is a schematic diagram of the first touch operation provided as an example.

[0032] Figure 8 This is an example of a first preprocessed image.

[0033] Figure 9 This is an example of a calibrated HSV image provided.

[0034] Figure 10 This is an example of a calibration mask image provided.

[0035] Figure 11 This is a second image provided as an example.

[0036] Figure 12 This is a schematic diagram of a second touch operation provided as an example.

[0037] Figure 13This is a second preprocessed image provided as an example.

[0038] Figure 14 This is a sample corrected image.

[0039] Figure 15 This is a sample test HSV image provided.

[0040] Figure 16 This is an example of a test mask image provided.

[0041] Figure 17 This is an example of a test mask image provided at the j+1th position.

[0042] Figure 18 This is an example morphological image provided.

[0043] Figure 19 This is a dynamic effect curve showing the x-coordinate of the center point of a test rectangle provided as an example.

[0044] Figure 20 This is a dynamic effect curve of the ordinate of the center point of a test rectangle provided as an example.

[0045] Figure 21 This is a dynamic effect curve of the height of a test rectangle provided as an example.

[0046] Figure 22 This is a dynamic effect curve for the width of a test rectangle provided as an example.

[0047] Figure 23 This is a sample grayscale image provided. Detailed Implementation

[0048] It should be noted that in the embodiments of this application, "several" refers to one or more, and "multiple" refers to two or more. The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.

[0049] It should also be noted that the methods disclosed in the embodiments of this application or the methods shown in the flowcharts include one or more steps for implementing the method. Without departing from the scope of the claims, the execution order of multiple steps can be interchanged, and some steps can also be deleted.

[0050] The terminal under test involved in the embodiments of this application can be an electronic device with a touch screen, such as a mobile phone, tablet computer, e-book reader, or personal digital assistant (PDA).

[0051] In this embodiment of the application, the testing device is used to test the smoothness of the display interface of the terminal under test. For example, as shown... Figure 1 As shown, the test device 10 may include a camera 110 and a control circuit (not shown). The camera 110 communicates with the control circuit and is used to capture images of the display interface of the terminal under test 20 within its field of view under the control of the control circuit, acquire captured images, and send the captured images to the control circuit. The control circuit is used to implement the test method of this application embodiment to determine the smoothness of the display interface of the terminal under test based on the captured images from the camera 110.

[0052] It is understood that the control circuit may include a central processing unit (CPU) and a graphics processing unit (GPU). The CPU is used to control the camera 110, and the GPU is used to implement the test method of this application embodiment to determine the smoothness of the display interface of the terminal under test 20 based on the images captured by the camera 110.

[0053] In other embodiments, the test device 10 may also include more components. For example, such as... Figure 2 As shown, the test device 10 also includes a robotic arm (not shown) and a drive circuit (not shown). The robotic arm includes an end effector 131, on which a stylus 132 can be mounted. For example, the end effector 131 can be a clamping mechanism for holding the stylus 132. The stylus 132 is used to perform touch operations on the touch screen of the terminal under test 20. Touch operations include clicking or dragging icons or controls on the display interface, swiping the screen to switch display interfaces, etc.

[0054] The drive circuit communicates with the control circuit and the robotic arm. The drive circuit generates drive signals under the control of the control circuit and sends drive signals to the robotic arm. The drive signals are used to drive the end effector 131 of the robotic arm to move along the target trajectory, thereby driving the stylus 132 to perform touch operations on the touch screen of the terminal under test 20. For example, the stylus 132 clicks the application icon on the display interface to launch the corresponding application.

[0055] It is understood that the drive circuit may include several transistors, such as MOS (Metal-Oxide-Semiconductor field-effect transistor) and IGBT (Insulated Gate Bipolar Transistor). The drive signal may be a pulse width modulation (PWM) signal. The drive signal is used to control the switching on and off of the transistors, thereby controlling the rotation speed and direction of the motor in the robotic arm, so as to drive the end effector 131 of the robotic arm to move along the target trajectory.

[0056] In another embodiment, such as Figure 3 As shown, the testing device 10 also includes a fixed bracket 150, on which a guide rail 151 is mounted. The length direction of the guide rail 151 is horizontal. A slider 111 is mounted on the camera 110. The slider 111 is assembled onto the guide rail 151 and can move along the length direction of the guide rail 151, thereby moving the camera 110 so that the field of view of the camera 110 can cover the display interface of the terminal under test. Figure 3 A robotic arm 130 is also shown, without an end effector. After the robotic arm 130 is fitted with an end effector, touch operations can be performed on the touchscreen of the terminal under test using a stylus mounted on the end effector. The test equipment 10 also includes an operating platform 160 for placing the terminal under test.

[0057] The following is based on Figure 3 The test method of this application embodiment is described in detail using the test equipment shown as an example.

[0058] like Figure 4 As shown, the test method includes the following steps:

[0059] S101, Obtain the first image.

[0060] The first image includes the display interface of the terminal under test and other parts besides the display interface, such as... Figure 5 The first image shown.

[0061] The display interface of the terminal under test (DUT) shows a calibration image, which is a pre-stored image on the DUT. The calibration image is used to calibrate the relative position between the DUT and camera 110. The calibration image includes M preset regions, where M is a positive integer. The color of each preset region is different from the background color of the calibration image. For example, as shown... Figure 6As shown, the calibration image includes preset regions 21-24, all of which are colored by a first color. The background color of the calibration image is a second color, and the first color and the second color are different.

[0062] It is understood that this embodiment does not limit the shape, size, or dimensions of the preset area in the calibration image.

[0063] In this embodiment, the first image is any frame from the calibration video, which is a video recorded by the camera 110. When the control circuit determines that the camera 110 is within the target location range, it sends a first shooting command to the camera 110 and a first driving command to the drive circuit. The first shooting command instructs the camera 110 to perform a shooting operation within a first time period. The first driving command instructs the drive circuit to generate a first driving signal, which drives the end effector of the robotic arm 130 to move along the first target trajectory. The target location range is the range within which the field of view of the camera 110 can cover the display interface of the terminal under test.

[0064] In response to the first shooting command, camera 110 captures the display interface of the terminal under test within its field of view during a first time period, obtains calibration video, and sends the calibration video frame by frame to the control circuit. The duration of the first time period can be set as needed.

[0065] During the first time period, the drive circuit responds to the first drive command, generates a first drive signal, and sends the first drive signal to the robotic arm 130. The end effector of the robotic arm 130 responds to the first drive signal, moves along a first target trajectory, and drives a stylus mounted on the end effector to perform a first touch operation on the touchscreen of the terminal under test. For example, as... Figure 7 As shown, the first touch operation includes clicking a thumbnail of the calibration image on the gallery application interface to fill the screen of the terminal under test with the calibration image.

[0066] It is understandable that during the first time period, camera 110 captures the display interface of the terminal under test, and the stylus performs the first touch operation on the touchscreen of the terminal under test. To reduce the occlusion of the display interface of the terminal under test by the stylus within the field of view of camera 110, constraints on the first target trajectory can be increased. When the end effector moves along the first target trajectory, the projected area of ​​the stylus on the display interface of the terminal under test is made smaller than the projection threshold, thereby reducing the area ratio of the stylus in the image captured by camera 110, making the stylus less noticeable in the captured image. The projection threshold can be set as needed.

[0067] S102. Preprocess the first image to obtain the first preprocessed image.

[0068] The first preprocessed image includes the display interface of the terminal under test, or may also include parts other than the display interface of the terminal under test, such as... Figure 8 The first preprocessed image shown.

[0069] The area ratio of the display interface of the terminal under test in the first image is smaller than the area ratio of the display interface of the terminal under test in the first preprocessed image. The area ratio of the display interface of the terminal under test in the image is the ratio of the pixel area of ​​the display interface to the pixel area of ​​the image. The pixel area of ​​the display interface is the product of the number of pixels contained in the display interface along the length direction of the image and the number of pixels contained in the display interface along the width direction of the image.

[0070] Preprocessing includes image cropping. The first image is cropped according to a first selected region to obtain a first preprocessed image. The size of the first preprocessed image is equal to the size of the first selected region. The size of the first selected region is greater than or equal to the size of the display interface of the terminal under test.

[0071] It is understandable that preprocessing can also include image enhancement, noise removal, geometric transformation, etc., to obtain a first preprocessed image that can meet different processing needs.

[0072] In other embodiments, step S102 may be omitted.

[0073] S103. Perform color space conversion on the first preprocessed image to obtain the calibrated HSV image.

[0074] In this embodiment, both the first image and the first preprocessed image are RGB (Red, Green, Blue) images. The first preprocessed image is converted from the RGB color space to the HSV (Hue, Saturation, Value) color space to obtain a calibrated HSV image, for example... Figure 9 The calibration HSV image shown.

[0075] For example, the cv2.cvtColor function in OpenCV, as shown in formula (1), is called to perform color space conversion.

[0076] hsv_image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV) (1)

[0077] Where hsv_image is the calibration HSV image, image is the first preprocessed image, and cv2.COLOR_BGR2HSV is the color space conversion code from RGB color space to HSV color space.

[0078] S104. Perform color filtering on the calibration HSV image according to the calibration target color filtering range to obtain the calibration mask image.

[0079] The calibration target color is the color of a preset region in the calibration image. The calibration mask image includes the calibration target color region and the portion excluding the calibration target color region. The calibration target color region is used to represent the preset region in the calibration image, for example... Figure 10 The calibration mask image shown.

[0080] For example, the target color filtering range is calibrated as shown in formula (2).

[0081]

[0082] Among them, lower bound To calibrate the lower limit of the target color filtering range in the HSV color space, upper bound To calibrate the upper limit of the target color filtering range in the HSV color space, H min To calibrate the minimum hue value corresponding to the target color filtering range, H max To calibrate the maximum hue value corresponding to the target color filtering range, S min To calibrate the minimum saturation value corresponding to the target color filtering range, S max To calibrate the maximum saturation value corresponding to the target color filtering range, V min To calibrate the minimum brightness value corresponding to the target color filtering range, V max This is used to determine the maximum brightness value corresponding to the target color filtering range.

[0083] Taking any pixel P on the calibration HSV image as an example, the pixel P on the calibration HSV image is converted to the corresponding pixel P' on the calibration mask image by formula (3).

[0084]

[0085] Where x is the number of pixels between pixel point P and the origin in the width direction of the calibration HSV image, and y is the number of pixels between pixel point P and the origin in the length direction of the calibration HSV image. The origin is the top-left vertex of the calibration HSV image. mask(x,y) is the pixel value of pixel point P' in the calibration mask image, lower bound ≤P(x,y)≤upper bound This indicates that the color of pixel P in the HSV image is designated as the target color. In masking, a pixel value of 1 (or True) indicates that pixel P is located in the target color region, and a pixel value of 0 (or False) indicates that pixel P is located in a non-target color region.

[0086] S105. Extract the contours of M calibrated target color regions from the calibration mask image.

[0087] For example, the findContours function in OpenCV, as shown in formula (4), is called to extract the contour of the calibrated target color region.

[0088] contours = findCoutours(mask, mode, method) (4)

[0089] Here, mask is the calibration mask image, mode is the contour retrieval mode, and method is the contour approximation method. For example, mode "RETR EXTERNAL" indicates retrieving the outermost contour, and method "CHAIN ​​APPROX SIMPLE" indicates retrieving the outermost contour.

[0090] S106. Obtain the M calibration rectangles corresponding to the outlines of the M calibration target color regions, and extract the center point coordinates of the M calibration rectangles.

[0091] In this embodiment, the calibration rectangle is the smallest bounding rectangle that defines the outline of the target color region.

[0092] For example, the minAreaRect function in OpenCV, as shown in formula (5), is called to obtain the calibration rectangle.

[0093] rect = minAreaRect(contour) (5)

[0094] Here, contour is the outline of the target color region, and rect is the calibration rectangle corresponding to the outline of the target color region.

[0095] When the `minAreaRect` function is called, it returns a structure of type `RotatedRect`, which contains the coordinates of the center point of the calibration rectangle, its width, height, and rotation angle. The center point coordinates of the calibration rectangle can be extracted by accessing the structure's `center` property.

[0096] S107. Calculate the perspective transformation matrix based on the coordinates of the center points of the M calibration rectangles in the calibration mask image and the coordinates of the center points of the M preset regions in the calibration image.

[0097] In this embodiment, the perspective transformation matrix is ​​used to represent the mapping relationship between the camera coordinate system and the coordinate system of the terminal under test. The coordinates of the center point of each calibration rectangle in the calibration mask image are the coordinates in the camera coordinate system, and the coordinates of the center point of each preset area in the calibration image are the coordinates in the coordinate system of the terminal under test.

[0098] For example, the perspective transformation matrix is ​​calculated by calling the getPerspectiveTransform function in OpenCV as shown in Equation (6).

[0099] perspective_matrix = getPerspectiveTransform(sorted_src_pts, dst_pts) (6)

[0100] Where sorted_src_pts is the set of coordinates of the center points of M preset regions in the calibration image, and dst_pts is the set of coordinates of the center points of M calibration rectangles in the calibration mask image.

[0101] S108. Obtain N frames of the second image.

[0102] Where N is a positive integer, and N>2. The second image of the camera 110 control circuit includes the display interface of the terminal under test and the part other than the display interface of the terminal under test.

[0103] The display interface of the terminal under test includes a target control area. The color of the target control area is different from the background color of the display interface, which is the color of the parts of the display interface excluding the target control area. For example, such as... Figure 11 As shown in the second image, the display interface of the terminal under test includes a target control area 25. The color of the target control area 25 is a third color, and the background color of the display interface is a fourth color. The third color and the fourth color are different.

[0104] The target control area is the area covered by the target control, which can be any interactive component in the user interface (UI). The target control can receive user input, display information, or control the application.

[0105] It is understood that this embodiment does not limit the shape, size, or dimensions of the target control area.

[0106] In this embodiment, the second image is a frame from a test video, which includes N frames of second images. The test video is recorded by camera 110 and is used to test the smoothness of the display interface of the terminal under test. When the control circuit determines that camera 110 is within the target location range, it sends a second shooting command to camera 110 and a second driving command to the drive circuit. The second shooting command instructs camera 110 to perform a shooting operation within a second time period. The second driving command instructs the drive circuit to generate a second driving signal, which drives the end effector of robotic arm 130 to move along the second target trajectory. The target location range is the range of the display interface of the terminal under test that can be covered by the field of view of camera 110.

[0107] In response to the second shooting command, camera 110 captures images of the display interface of the terminal under test within its field of view during a second time period, obtaining test video, and sends the test video frame by frame to the control circuit. The duration of the second time period can be set as needed. The second time period follows the first time period.

[0108] During the second time period, the drive circuit responds to the second drive command, generates a second drive signal, and sends the second drive signal to the robotic arm 130. The end effector of the robotic arm 130 responds to the second drive signal, moves along the second target trajectory, and drives the stylus mounted on the end effector to perform a second touch operation on the touchscreen of the terminal under test. For example, taking a gallery application as an example, such as... Figure 12 As shown, the second touch operation involves clicking the Gallery app icon on the Home screen, causing the Gallery app interface to fill the screen of the device under test. On the Home screen, the area covered by the Gallery app icon is the target control area. On the Gallery app interface, the entire Gallery app interface is the target control area.

[0109] It is understandable that during the second time period, camera 110 captures the display interface of the terminal under test, and the stylus performs a second touch operation on the touchscreen of the terminal under test. To reduce the occlusion of the display interface of the terminal under test by the stylus within the field of view of camera 110, constraints on the second target trajectory can be increased. When the end effector moves along the second target trajectory, the projected area of ​​the stylus on the display interface of the terminal under test is made smaller than the projection threshold, thereby reducing the area ratio of the stylus in the image captured by camera 110, making the stylus less noticeable in the captured image.

[0110] S109. Preprocess the N frames of the second image to obtain N frames of the second preprocessed image.

[0111] The second preprocessed image includes the display interface of the terminal under test, or may also include parts other than the display interface of the terminal under test, such as... Figure 13The second preprocessed image shown.

[0112] The area ratio of the display interface of the terminal under test in the second image is smaller than the area ratio of the display interface of the terminal under test in the second preprocessed image.

[0113] Preprocessing includes image cropping. The second image is cropped according to a second selected region to obtain a second preprocessed image. The size of the second preprocessed image is equal to the size of the second selected region. The size of the second selected region is greater than or equal to the size of the display interface of the terminal under test.

[0114] It is understandable that preprocessing can also include image enhancement, noise removal, geometric transformation, etc., to obtain a second preprocessed image that can meet different processing needs.

[0115] In other embodiments, step S109 may be omitted.

[0116] S110. The position of the N frames of the second preprocessed image is corrected by the perspective transformation matrix to obtain the N frames of corrected images.

[0117] In this embodiment, the calibration image includes the display interface of the terminal under test, for example... Figure 14 The corrected image is shown. Position correction of the second preprocessed image involves transforming the second preprocessed image from the camera coordinate system to the coordinate system of the terminal under test.

[0118] Taking any pixel P'(x',y') in the second preprocessed image as an example, the pixel P' in the second preprocessed image is mapped to the corresponding pixel P(x,y) in the corrected image using formula (7).

[0119]

[0120] Where x represents the number of pixels between pixel point P and the origin in the image width direction of the terminal under test coordinate system, and y represents the number of pixels between pixel point P and the origin in the image length direction of the terminal under test coordinate system. The origin of the terminal under test coordinate system is the top-left vertex of the image. x' represents the number of pixels between pixel point P' and the origin in the image width direction of the camera coordinate system, and y' represents the number of pixels between pixel point P' and the origin in the image length direction of the camera coordinate system. The origin of the camera coordinate system is the top-left vertex of the image. H -1 It is the inverse of the perspective transformation matrix.

[0121] S111. Perform color space conversion on the N frames of corrected images to obtain N frames of test HSV images.

[0122] In this embodiment, the second image, the second preprocessed image, and the corrected image are all RGB images. The corrected image is converted from the RGB color space to the HSV color space to obtain the test HSV image, for example... Figure 15 The test HSV image shown.

[0123] For details on color space conversion, please refer to the relevant description in step S103, which will not be repeated here.

[0124] It is understandable that step S110 can also be executed after S111 is completed.

[0125] S112. Perform color filtering on N frames of test HSV images according to the color filtering range of the j-th test target to obtain N frames of the j-th test mask image.

[0126] Where j is a positive integer. The j-th test target color is the background color of the test HSV image, that is, the color of the part of the test HSV image excluding the target control area. The test mask image contains the j-th test target color region and the part excluding the j-th test target color region, for example... Figure 16 The test mask image shown.

[0127] For details on color filtering, please refer to the relevant description in step S104, which will not be repeated here.

[0128] S113. Calculate the average and standard deviation of hue, saturation, and brightness of the j-th test target color region in the j-th test mask image of N frames.

[0129] For example, for the set of pixels in the j-th test target color region, the average value and standard deviation of hue, saturation, and brightness are calculated using formulas (8)-(10).

[0130]

[0131] Where (x,y) represents the coordinates of any pixel in the pixel set P of the j-th test target color region, x represents the number of pixels between the pixel and the origin in the length direction of the j-th test mask image, y represents the number of pixels between the pixel and the origin in the width direction of the j-th test mask image, and the origin is the top-left vertex of the j-th test mask image. |P| represents the number of pixels in the pixel set P, H(x,y) represents the hue value corresponding to the pixel coordinates (x,y), and μ H σ represents the average hue of the j-th test target color region. H Let be the standard deviation of the hue of the j-th test target color region.

[0132]

[0133] Where S(x,y) is the saturation value corresponding to the pixel coordinates (x,y), and μ S Let σ be the average saturation value of the j-th test target color region. S Let be the standard deviation of saturation in the j-th test target color region.

[0134]

[0135] Where V(x,y) is the brightness value corresponding to the pixel coordinates (x,y), and μ V Let σ be the average brightness of the j-th test target color region. V Let be the standard deviation of the lightness of the j-th test target color region.

[0136] S114. Set the filter range for the (j+1)th test target color based on the adjustment parameters and the average and standard deviation of the hue, saturation, and brightness of the j-th test target color region.

[0137] The adjustment parameters include hue adjustment parameters, saturation adjustment parameters, and brightness adjustment parameters. The filtering range of the (j+1)th test target color is smaller than the filtering range of the jth test target color.

[0138] For example, the color filtering range of the (j+1)th test target is set by formula (11).

[0139]

[0140] Where, k H k is the hue adjustment parameter. S k is the saturation adjustment parameter. V k is the brightness adjustment parameter. H k S k V All are greater than 0. H min H represents the minimum hue value corresponding to the (j+1)th test target color filtering range. max S represents the maximum hue value corresponding to the (j+1)th test target color filtering range. min S represents the minimum saturation value corresponding to the (j+1)th test target color filtering range. max V represents the maximum saturation value corresponding to the (j+1)th test target color filtering range. min V represents the minimum brightness value corresponding to the (j+1)th test target color filtering range. max This represents the maximum brightness value corresponding to the (j+1)th test target color filtering range.

[0141] S115. Perform color filtering on the N frames of test HSV images according to the color filtering range of the (j+1)th test target to obtain the (j+1)th test mask image of N frames.

[0142] The (j+1)th test target color is the background color of the test HSV image. The (j+1)th test mask image contains the (j+1)th test target color region and the rest of the image. In the (j+1)th test mask image, the (j+1)th test target color region is white (pixel value 1), and the rest of the image is black (pixel value 0), for example... Figure 17 The image shown is the (j+1)th test mask. The size of the (j+1)th test target color region is smaller than the size of the jth test target color region.

[0143] For details on color filtering, please refer to the relevant description in step S104, which will not be repeated here.

[0144] In this embodiment, the hue, saturation, and brightness values ​​of the test HSV image are significantly affected by ambient lighting and color. The j-th test target color has a wide filtering range. By setting the adjustment parameters corresponding to hue, saturation, and brightness to set the filtering range of the (j+1)-th test target color, the accuracy of color filtering can be improved. Furthermore, the adjustment parameters corresponding to hue, saturation, and brightness can be flexibly set, thereby enhancing the flexibility of color filtering. Because the color filtering range is reduced, the size of the test target color region is reduced, thus reducing the influence of ambient lighting and color on the test target color region in the test HSV image, thereby enhancing the robustness of the test target color region under different ambient lighting and color conditions.

[0145] In other embodiments, the test HSV image can be color filtered multiple times to extract a more accurate test target color region. For example, after obtaining the (j+1)th test mask image, the average and standard deviation of the hue, saturation, and brightness of the (j+1)th test target color region in the (j+1)th test mask image are calculated. Then, based on the adjustment parameters and the average and standard deviation of the hue, saturation, and brightness of the (j+1)th test target color region, a (j+2)th test target color filtering range is set. Finally, the test HSV image is color filtered according to the (j+2)th test target color filtering range.

[0146] S116. Perform morphological processing on the (j+1)th test mask image of N frames to obtain N frames of morphological images.

[0147] The morphological image includes the (j+1)th test target color region and the portion excluding the (j+1)th test target color region, for example... Figure 18 The morphological image shown.

[0148] Morphological processing is used to eliminate noise, fill holes, and connect disconnected test target color regions (regions j+1). Morphological processing includes using a closing operation to smooth the boundaries of the (j+1)th test target color region, fill holes, and connect adjacent (j+1)th test target color regions. The closing operation first dilates the (j+1)th mask image, then erodes the dilated image. The dilation operation slides a structuring element across the (j+1)th mask image, assigning the maximum pixel value from all pixels covered by the structuring element to the pixel located at the center of the structuring element, thus increasing the boundary of the (j+1)th test target color region. The erosion operation slides a structuring element across the dilated image, assigning the minimum pixel value from all pixels covered by the structuring element to the pixel located at the center of the structuring element, thus decreasing the boundary of the (j+1)th test target color region. The structuring element can be rectangular or circular, and its size can be customized.

[0149] For example, morphological processing is performed using formula (12).

[0150]

[0151] Here, A is the (j+1)th mask image, and B is the structuring element, such as a 17*17 rectangular structuring element. `Close(A,B)` performs a closing operation on the (j+1)th mask image A using the structuring element B. `Dilate(A,B)` performs a dilation operation on the (j+1)th mask image A using the structuring element B. `Erode(Dilate(A,B),B)` performs an erosion operation on the dilated image using the structuring element B.

[0152] In other embodiments, steps S113-S115 may be omitted. That is, morphological processing can be performed on the N frames of test mask images to obtain N frames of morphological images.

[0153] S117. Extract the contours of N target control regions from N frames of morphological images.

[0154] For details on how to extract the contour, please refer to the relevant description of step S105, which will not be repeated here.

[0155] In other embodiments, step S116 may be omitted. That is, the contours of N target control regions can be extracted from the j-th test mask image of N frames or the (j+1)-th test mask image of N frames.

[0156] S118. Obtain N test rectangles corresponding to the outlines of N target control areas, and extract the center point coordinates, height, and width of the N test rectangles.

[0157] In this embodiment, the test rectangle is the smallest bounding rectangle of the outline of the target control area.

[0158] For example, the boundingRect function in OpenCV, as shown in formula (13), is called to obtain the test rectangle corresponding to the outline of the target control, and the height and width of the test rectangle are extracted.

[0159] boundingRect(C t ) = (x, y, w, h) (13)

[0160] Among them, C t The outline of the target control is defined by (x, y), where (x, y) are the coordinates of the top-left vertex of the test rectangle, w is the width of the test rectangle, and h is the height of the test rectangle. x represents the number of pixels between the top-left vertex of the test rectangle and the origin in the width direction of the morphological image, and y represents the number of pixels between the top-left vertex of the test rectangle and the origin in the length direction of the morphological image. The origin is the top-left vertex of the morphological image.

[0161] The `boundingRect` function returns a tuple when called, which stores multiple ordered, immutable data items. In this embodiment, the tuple includes the coordinates of the top-left vertex of the test rectangle, the width of the test rectangle, and its height.

[0162] The coordinates of the center point of the test rectangle are extracted using formula (14).

[0163]

[0164] Among them, (x c ,y c ) represents the coordinates of the center point of the test rectangle.

[0165] S119. Based on the center point coordinates, height, and width of the N test rectangles, draw the dynamic effect curves of the center point x-coordinate, the center point y-coordinate, the height, and the width, respectively.

[0166] In this embodiment, the dynamic effects of target controls on the display interface of the terminal under test 20 are used to characterize the smoothness of the display interface. The dynamic effects of the target controls are represented by the changing presentation state of the target controls over time. The presentation state of the target controls can include the displacement, posture, and size of the target controls. For example, within a preset time period, the target controls move from a first position to a second position, or the target controls rotate by a predetermined angle, or the target controls scale by a predetermined ratio.

[0167] Taking the target control as an example of the application displayed on the terminal under test, the dynamic effect of the target control starts from clicking the application icon and ends when the application interface fills the screen. Figure 19 It is a dynamic effect curve of the x-coordinate of the center point of the test rectangle, which shows how the size of the x-coordinate of the center point of the test rectangle changes with the number of test video frames. Figure 20 It is a dynamic effect curve of the ordinate of the center point of the test rectangle, which shows how the size of the ordinate of the center point of the test rectangle changes with the number of test video frames. Figure 21 It is a dynamic effect curve of the height of the test rectangle, which shows how the height of the test rectangle changes with the number of test video frames. Figure 22 It is a dynamic effect curve of the width of the test rectangle, which shows how the width of the test rectangle changes with the number of test video frames.

[0168] S120. Perform grayscale transformation on the N frames of corrected images to obtain N frames of grayscale images.

[0169] In this embodiment, the calibration image is an RGB image. The grayscale image contains the display interface of the terminal under test, and the display interface includes the target control area, for example... Figure 23 The grayscale image shown.

[0170] For example, the cv2.cvtColor function in OpenCV, as shown in formula (15), is called to perform grayscale transformation.

[0171] gray_image = cv2.cvtColor(corrected_image, cv2.COLOR_BGR2GRAY) (15)

[0172] Where gray_image is a grayscale image, corrected_image is a corrected image, and cv2.COLOR_BGR2GRAY is the color space conversion code from RGB color space to grayscale space.

[0173] S121. Traverse N frames of grayscale images, calculate the grayscale difference between the region of interest in the (i+1)th frame and the region of interest in the i-th frame, and obtain N-1 grayscale difference values ​​after traversal.

[0174] Where i is a positive integer, and 1≤i≤N-1.

[0175] In this embodiment, the Region of Interest (ROI) in the grayscale image is a preset rectangular region, which can be represented as ROI(x,y,w,h), where (x,y) are the coordinates of the top-left vertex of the ROI, w is the width of the ROI, and h is the height of the ROI. The setting of the ROI in each frame of the grayscale image is independent of each other. That is, the ROI in two frames of grayscale images can be the same or different.

[0176] For example, the grayscale difference between the ROI in the (i+1)th grayscale image and the ROI in the ith grayscale image is calculated using formula (16).

[0177]

[0178] in, Let be the grayscale difference between the ROI in the (t+1)th frame and the ROI in the tth frame, where t is the frame number. Let be the grayscale value of the ROI in the t-th frame of the grayscale image. Let (x, y) be the grayscale value of the ROI in the (t+1)th frame of the grayscale image, and (x, y) be the pixel point located within the ROI.

[0179] S122. Determine the start frame and end frame of the dynamic effect based on N-1 grayscale differences.

[0180] In this embodiment, the starting frame of the dynamic effect is a grayscale image when the target control is triggered, and the ending frame of the dynamic effect is a grayscale image when the presentation state of the target control tends to stabilize.

[0181] Taking the application displayed on the terminal under test as the target control as an example, the dynamic effect of the target control starts from clicking the application icon and ends when the application interface fills the screen. The starting frame of the dynamic effect is a grayscale image when the application icon is clicked, and the ending frame of the dynamic effect is a grayscale image when the application interface fills the screen. In the starting frame of the dynamic effect, the ROI is the application icon area, and in the ending frame of the dynamic effect, the ROI is the screen edge area.

[0182] It's understandable that the grayscale difference corresponding to the ROI in the starting frame of the dynamic effect is larger, while the grayscale difference corresponding to the ROI in the ending frame of the dynamic effect is smaller. The grayscale difference corresponding to the ROI in the starting frame of the dynamic effect is the difference between the grayscale of the ROI in the starting frame and the ROI in the next frame of the grayscale image. The grayscale difference corresponding to the ROI in the ending frame of the dynamic effect is the difference between the grayscale of the ROI in the ending frame and the ROI in the previous frame of the grayscale image.

[0183] For example, the start frame and end frame of the dynamic effect are determined by formula (17).

[0184]

[0185] Among them, t start t is the frame number of the starting frame for the dynamic effect. end τ is the frame number at which the dynamic effect ends. start τ is the grayscale threshold of the starting frame. end The grayscale threshold is set to the end frame grayscale threshold. The starting frame for the dynamic effect is the grayscale image with the smallest frame count among several grayscale images whose grayscale difference corresponding to the ROI is greater than the starting frame grayscale threshold. The ending frame for the dynamic effect is the grayscale image with the smallest frame count among several grayscale images whose grayscale difference corresponding to the ROI is less than the ending frame grayscale threshold. The starting and ending frame grayscale thresholds can be set as needed.

[0186] Determining the start and end frames of a dynamic effect based on N-1 grayscale differences involves: traversing the N-1 grayscale differences; if the i-th grayscale difference is greater than the grayscale threshold of the start frame, the i-th grayscale image is determined as the target start frame; if the i-th grayscale difference is less than the grayscale threshold of the end frame, the i-th grayscale image is determined as the target end frame. After traversal, K target start frames and T target end frames are obtained. The grayscale image with the smallest frame count among the K target start frames is determined as the start frame of the dynamic effect, and the grayscale image with the smallest frame count among the T target end frames is determined as the end frame of the dynamic effect. The i-th grayscale difference is the grayscale difference between the region of interest in the (i+1)-th grayscale image and the region of interest in the i-th grayscale image. Here, K and T are both positive integers, and 1 ≤ K ≤ N-1, 1 ≤ T ≤ N-1.

[0187] S123. Calculate the duration of the dynamic effect based on the frame number of the starting frame of the dynamic effect, the frame number of the ending frame of the dynamic effect, and the frame rate of the test video.

[0188] In this embodiment, the duration of the dynamic effect is the ratio of the frame difference to the frame rate of the test video, and the frame difference is the difference between the number of frames in the end frame of the dynamic effect and the number of frames in the start frame of the dynamic effect.

[0189] For example, the duration of the dynamic effect is calculated using formula (18).

[0190]

[0191] Where T is the duration of the dynamic effect, f is the frame rate of the test video, and t start t is the frame number of the starting frame for the dynamic effect. end The number of frames at the end of the dynamic effect.

[0192] S124. When the dynamic effect curves of the center point x-coordinate, the center point y-coordinate, the height, and the width all meet the smoothness condition, and the duration of the dynamic effect is less than or equal to the duration threshold, the display interface is determined to be smooth.

[0193] Among them, the smoothness condition of the dynamic effect curve includes that the variance of all point values ​​on the dynamic effect curve is less than or equal to the variance threshold, or the standard deviation of all point values ​​is less than or equal to the standard deviation threshold, or the mean absolute difference (MAD) between each adjacent point value is less than or equal to the MAD threshold, or the mean squared difference (MSD) between each adjacent point value is less than or equal to the MSD threshold, or the deviation between the dynamic effect curve and the fitted curve is within the deviation range.

[0194] Understandably, the duration threshold, variance threshold, standard deviation threshold, MAD threshold, MSD threshold, and deviation range can all be set as needed.

[0195] In this embodiment, because the contour extraction algorithm is susceptible to environmental noise, color, and lighting, the contour extraction error may be large, resulting in low accuracy. Therefore, by converting grayscale and using the frame difference method to calculate the grayscale difference between two consecutive grayscale images, the duration of the dynamic effect can be calculated based on the grayscale difference. This effectively overcomes the interference of contour extraction error on the calculation of the dynamic effect duration, thereby accurately identifying the start and end frames of the dynamic effect and improving testing accuracy. Moreover, this embodiment can effectively detect various dynamic effects of target controls, such as sliding icons or windows, opening or closing applications, etc., and has a wide range of applications.

[0196] In another embodiment, step S119 may be omitted. That is, when the arrays of the x-coordinates, y-coordinates, heights, and widths of the center points of the N test rectangles all satisfy the smoothness condition, and the duration of the dynamic effect is less than or equal to the duration threshold, the display interface is determined to be smooth. The smoothness condition for the arrays includes that the variance of all values ​​in the array is less than or equal to the variance threshold, or the standard deviation of all values ​​is less than or equal to the standard deviation threshold, or the MAD between adjacent values ​​is less than or equal to the MAD threshold, or the MSD between adjacent values ​​is less than or equal to the MSD threshold.

[0197] In another embodiment, steps S120-S123 may be omitted. That is, the display interface is determined to be smooth when the dynamic effect curves of the center point's x-coordinate, center point's y-coordinate, height, and width all meet the smoothness condition. Alternatively, the display interface is determined to be smooth when the arrays of the center point's x-coordinates, center point's y-coordinates, height, and width of the N test rectangles all meet the smoothness condition.

[0198] In another embodiment, step S118 can involve extracting only the center point coordinates of the N test rectangles and using these coordinates to evaluate the smoothness of the display interface. Alternatively, it can be combined with steps S120-S123, using the center point coordinates of the N test rectangles and the duration of the dynamic effect to evaluate the smoothness of the display interface. For example, if the array of center point coordinates of the N test rectangles meets the smoothness condition, the display interface is determined to be smooth. Alternatively, dynamic effect curves for the horizontal and vertical coordinates of the center points can be plotted based on the center point coordinates of the N test rectangles. If both curves meet the smoothness condition, the display interface is determined to be smooth. Or, if the array of center point coordinates of the N test rectangles meets the smoothness condition and the duration of the dynamic effect is less than or equal to a duration threshold, the display interface is determined to be smooth. Alternatively, based on the center point coordinates of N test rectangles, plot the dynamic effect curves of the x-coordinate and y-coordinate of the center point. If both the dynamic effect curves of the x-coordinate and y-coordinate of the center point meet the smoothness condition, and the duration of the dynamic effect is less than or equal to the duration threshold, the display interface is considered smooth.

[0199] The above embodiments are based on Figure 3 The test method is explained in detail using the test device 10 shown as an example. It can be understood that the test method can also be applied to... Figure 1 or Figure 2 The test device 10 shown can also be applied to computer-readable storage media and computer program products.

[0200] A computer-readable storage medium stores computer instructions that, when executed by a processor, implement the test method of the embodiments of this application.

[0201] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program circuits or other data). Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tapes, disk storage or other magnetic storage devices.

[0202] The computer program product includes computer instructions that, when executed by a processor, implement the test methods of the embodiments of this application.

[0203] The embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of this application.

Claims

1. A method for testing the smoothness of a display interface, applied to a testing device, characterized in that, The method includes: Obtain N test RGB images, each containing the display interface of the terminal under test. The display interface includes a target control area, the color of which is different from the background color of the display interface; N is a positive integer and N>2; The N-frame test RGB images located in the camera coordinate system are converted into N-frame correction images located in the coordinate system of the terminal under test. Convert the N frames of corrected images into N frames of test HSV images; Color filtering is performed on the N test HSV images to obtain N test mask images; Obtain N test rectangles corresponding to the contours of N target control regions in the N frame test mask images, wherein the test rectangles are the minimum bounding rectangles of the contours of the target control regions; If the center point coordinates of the N test rectangles meet the smoothness condition, the display interface of the terminal under test is determined to be smooth.

2. The method for testing the smoothness of the display interface as described in claim 1, characterized in that, The method further includes: Perform grayscale transformation on the N frames of corrected images to obtain N frames of grayscale images; Traverse the N frames of grayscale images, calculate the grayscale difference between the region of interest in the (i+1)th frame and the region of interest in the i-th frame; after traversal, obtain N-1 grayscale difference values; i is a positive integer, and 1≤i≤N-1; The start frame and end frame of the dynamic effect are determined based on the N-1 grayscale differences. The duration of the dynamic effect is calculated based on the frame number of the starting frame of the dynamic effect, the frame number of the ending frame of the dynamic effect, and the frame rate of the test video. When the center point coordinates of the N test rectangles meet the smoothness condition, and the duration of the dynamic effect is less than or equal to the duration threshold, the display interface of the terminal under test is determined to be smooth.

3. The method for testing the smoothness of the display interface as described in claim 2, characterized in that, The step of determining the start frame and end frame of the dynamic effect based on the N-1 grayscale differences includes: Traverse the N-1 grayscale differences. When the i-th grayscale difference is greater than the grayscale threshold of the starting frame, determine the i-th grayscale image as the target starting frame; when the i-th grayscale difference is less than the grayscale threshold of the ending frame, determine the i-th grayscale image as the target ending frame. After traversal, obtain K target starting frames and T target ending frames. The i-th grayscale difference is the grayscale difference between the region of interest in the (i+1)-th grayscale image and the region of interest in the i-th grayscale image. K and T are both positive integers, and 1≤K≤N-1, 1≤T≤N-1. The grayscale image with the smallest frame count among the K target starting frames is determined as the starting frame of the dynamic effect; The grayscale image with the smallest frame count among the T target end frames is determined as the end frame of the dynamic effect.

4. The method for testing the smoothness of the display interface as described in claim 1, characterized in that, The method further includes: If the center point coordinates, height, and width of the N test rectangles all meet the smoothness condition, the display interface of the terminal under test is determined to be smooth.

5. The method for testing the smoothness of the display interface as described in claim 2 or 3, characterized in that, The method further includes: When the center point coordinates, height, and width of the N test rectangles all meet the smoothness condition, and the duration of the dynamic effect is less than or equal to the duration threshold, the display interface of the terminal under test is determined to be smooth.

6. The method for testing the smoothness of the display interface as described in claim 4 or 5, characterized in that, The center point coordinates, height, and width of the N test rectangles all satisfy the following smoothness conditions: The dynamic effect curves corresponding to the center point coordinates, height, and width of the N test rectangles all satisfy the following: The variance of all point values ​​is less than or equal to the variance threshold, or The standard deviation of all point values ​​is less than or equal to the standard deviation threshold, or The average absolute difference between adjacent point values ​​is less than or equal to the average absolute difference threshold, or The mean squared difference between adjacent point values ​​is less than or equal to the mean squared difference threshold, or The deviation between the dynamic effect curve and the fitted curve is within the deviation range.

7. The method for testing the smoothness of the display interface as described in claim 4 or 5, characterized in that, The center point coordinates, height, and width of the N test rectangles all satisfy the following smoothness conditions: The arrays corresponding to the center point coordinates, height, and width of the N test rectangles all satisfy the following: The variance of all values ​​is less than or equal to the variance threshold, or The standard deviation of all values ​​is less than or equal to the standard deviation threshold, or The average absolute difference between each adjacent value is less than or equal to the average absolute difference threshold, or The mean squared difference between each adjacent value is less than or equal to the mean squared difference threshold.

8. The method for testing the smoothness of the display interface as described in any one of claims 1-7, characterized in that, The step of color filtering the N frames of test HSV images to obtain N frames of test mask images includes: The N test HSV images are color filtered according to the color filtering range of the j-th test target to obtain N frame j-th test mask images, where j is a positive integer; Calculate the average and standard deviation of the hue, saturation, and brightness of the j-th test target color region in the j-th test mask image of the N frames; The filter range for the (j+1)th test target color is set according to the adjustment parameters and the average and standard deviation of the hue, saturation, and brightness of the j-th test target color region; the filter range for the (j+1)th test target color is smaller than the filter range for the j-th test target color; the adjustment parameters include hue adjustment parameters, saturation adjustment parameters, and brightness adjustment parameters; Color filtering is performed on the N frames of test HSV images according to the (j+1)th test target color filtering range to obtain the (j+1)th test mask image of N frames.

9. The method for testing the smoothness of the display interface as described in any one of claims 1-8, characterized in that, Before converting the N-frame test RGB images located in the camera coordinate system into N-frame corrected images located in the coordinate system of the terminal under test, the method further includes: A calibration RGB image is acquired, the calibration RGB image includes the display interface of the terminal under test, the display interface displays the calibration image, the calibration image includes M preset regions, M is a positive integer; the color of the preset regions is different from the background color of the calibration image; Convert the calibrated RGB image into a calibrated HSV image; Color filtering is performed on the calibration HSV image to obtain a calibration mask image; Obtain M calibration rectangles corresponding to the contours of M calibration target color regions in the calibration mask image, wherein the calibration rectangles are the minimum bounding rectangles of the contours of the calibration target color regions; The perspective transformation matrix is ​​calculated based on the center point coordinates of the M calibration rectangles and the center point coordinates of the M preset regions in the calibration image. The perspective transformation matrix is ​​used to convert the N-frame test RGB image located in the camera coordinate system into an N-frame correction image located in the coordinate system of the terminal under test.

10. The method for testing the smoothness of the display interface as described in any one of claims 1-9, characterized in that, The step of obtaining N test rectangles corresponding to the contours of N target control regions in the N frame test mask images includes: The N frames of test mask images are subjected to morphological processing to obtain N frames of morphological images; Extract the contours of N target control regions from the N frames of morphological images; Obtain N test rectangles corresponding to the outlines of the N target control areas.

11. The method for testing the smoothness of the display interface as described in claim 9, characterized in that, The step of obtaining the M calibration rectangles corresponding to the contours of the M calibration target color regions in the calibration mask image includes: Extract the contours of M calibrated target color regions from the calibration mask image; Obtain M calibration rectangles corresponding to the contours of the M calibration target color regions.

12. A testing device, characterized in that, The testing equipment includes a camera and a control circuit, and the camera is communicatively connected to the control circuit. The camera is used to capture images of the display interface of the terminal under test within its field of view under the control of the control circuit, acquire captured images, and send the captured images to the control circuit; the captured images include N frames of test RGB images, the test RGB images contain the display interface of the terminal under test, the display interface includes a target control area, the color of the target control area is different from the background color of the display interface; N is a positive integer, and N>2; The control circuit is used to acquire the N-frame test RGB images, convert the N-frame test RGB images located in the camera coordinate system into N-frame corrected images located in the coordinate system of the terminal under test, and convert the N-frame corrected images into N-frame test HSV images. Color filtering is performed on the N-frame test HSV images to obtain N-frame test mask images; N test rectangles are obtained corresponding to the contours of N target control regions in the N-frame test mask images, and the test rectangles are the minimum bounding rectangles of the contours of the target control regions; when the center point coordinates of the N test rectangles meet the smoothness condition, the display interface of the terminal under test is determined to be smooth.

13. The testing equipment as described in claim 12, characterized in that, The testing equipment also includes a robotic arm and a drive circuit. The robotic arm includes an end effector for mounting a stylus, which is used to perform touch operations on the touch screen of the terminal under test. The drive circuit is electrically connected to the control circuit and the robotic arm. The drive circuit is used to generate a drive signal under the control of the control circuit and to send a drive signal to the robotic arm. The drive signal is used to drive the end effector of the robotic arm to move along the target trajectory, thereby driving the stylus to perform touch operation on the touch screen of the terminal under test.

14. The testing equipment as described in claim 12 or 13, characterized in that, The testing equipment also includes a fixed bracket with a guide rail mounted on it and a slider mounted on the camera. The slider is used to assemble onto the guide rail. When the slider moves along the length of the guide rail, it moves the camera so that the camera's field of view can cover the display interface of the terminal under test. The length of the guide rail is horizontal.

15. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the processor executes the computer instructions, it implements the method for testing the smoothness of the display interface as described in any one of claims 1-11.

Images