Touch screen test method, device and equipment based on visual calibration, and storage medium

Abstract

The application discloses a touch screen test method, device and equipment based on visual calibration, and a storage medium. The method comprises the following steps: controlling a camera to capture a to-be-tested screen, and acquiring a first region of interest in a picture and a second region of interest corresponding to a touch end of a mechanical arm; acquiring a target touch position and mapping the target touch position into a target pixel coordinate in the picture of the camera; calculating a normalized relative position according to the target pixel coordinate and a boundary of the first region of interest; calculating a target coordinate of the mechanical arm according to the normalized relative position and a boundary of the second region of interest; controlling the mechanical arm to move to the target coordinate to perform a touch operation; acquiring a current picture of the screen after the touch operation through the camera to perform visual verification, and confirming whether an expected response is generated. The application realizes coordinate decoupling through hierarchical calibration, improves calibration flexibility and positioning accuracy, and constructs a visual closed loop to realize unmanned and automatic test.

Description

Technical Field

[0001] This invention relates to the technical field of automated testing, and in particular to a touchscreen testing method, apparatus, device, and storage medium based on visual calibration. Background Technology

[0002] With the rapid iteration of smart cockpit technology, automated testing technology for in-vehicle touchscreens is receiving increasing attention from the industry. Current related technologies are mainly divided into software touch injection solutions based on system debugging interfaces and touch simulation solutions based on physical robotic arms. While software injection solutions offer higher execution efficiency, they have inherent limitations in reproducing physical characteristics such as touch timing and pressure sensitivity, and are highly dependent on the system-level debugging permissions of the device under test. Therefore, physical robotic arm touch solutions combined with machine vision are gradually becoming the mainstream development direction. This technology uses a camera to capture the screen under test, establishes a mapping relationship between image pixels and the physical coordinates of the robotic arm, and then guides the end effector of the robotic arm to complete interactive actions such as clicking and swiping, objectively reflecting the true physical response characteristics of the screen.

[0003] Existing robotic arm touchscreen testing technologies still have some shortcomings in coordinate calibration and testing verification. On the one hand, existing coordinate calibration mostly adopts a single-layer direct mapping mechanism, which does not independently decouple and hierarchically manage the screen display area within the camera's field of view from the actual physical working range of the robotic arm. When the camera pose is finely adjusted or the device under test is replaced, a time-consuming and inflexible global recalibration is required. Furthermore, simple linear mapping is difficult to effectively absorb edge positioning errors and slight perspective distortions. On the other hand, existing technologies generally lack visual feedback closed-loop verification. Their visual acquisition is mostly used only for offline video recording. After the robotic arm performs a touch operation, it fails to use visual algorithms to compare and analyze the current screen image in real time, and cannot automatically confirm whether the touch action triggers the expected interface response. As a result, the test results still heavily rely on manual frame-by-frame verification in the later stages, which seriously restricts the realization of unattended automated testing. Summary of the Invention

[0004] To overcome the shortcomings of existing technical solutions, embodiments of the present invention provide a touch screen testing method, apparatus, device, and storage medium based on visual calibration.

[0005] The technical solution adopted by this invention to solve its technical problem is: In a first aspect, the present invention provides a touchscreen testing method based on visual calibration, wherein the touchscreen testing method is applied to a control terminal, the control terminal being connected to a robotic arm and a camera control, and the touchscreen testing method includes: The camera is controlled to capture images of the screen under test, and a first region of interest is obtained from the image captured by the camera, and a second region of interest corresponding to the first region of interest is obtained when the touch end of the robotic arm touches the display area of ​​the screen under test. The target touch position of the robotic arm's touch end on the display interface is obtained, and the target touch position is mapped to the target pixel coordinates in the image captured by the camera; Calculate the normalized relative position based on the target pixel coordinates and the boundary of the first region of interest; The target coordinates of the robotic arm are calculated based on the normalized relative position and the boundary of the second region of interest; Control the touch end of the robotic arm to move to the target coordinates to perform a touch operation; The camera captures the current image of the screen under test after the touch operation is performed by the touch terminal of the robotic arm, and performs visual verification on the current image of the screen under test to confirm whether the touch operation produces the expected interface response.

[0006] Secondly, the present invention provides a touchscreen testing device based on visual calibration, wherein the touchscreen testing device is disposed in a control terminal, the control terminal being connected to a robotic arm and a camera control, and the touchscreen testing device includes: The acquisition module is used to control the camera to capture images of the screen under test, and to acquire a first region of interest in the image captured by the camera, and a second region of interest corresponding to the first region of interest when the touch end of the robotic arm touches the display area of ​​the screen under test. The first mapping module is used to obtain the target touch position of the touch end of the robotic arm on the display interface, and map the target touch position to the target pixel coordinates in the image captured by the camera; The calculation module is used to calculate the normalized relative position based on the target pixel coordinates and the boundary of the first region of interest; The second mapping module is used to calculate the target coordinates of the robotic arm based on the normalized relative position and the boundary of the second region of interest; The control module is used to control the touch end of the robotic arm to move to the target coordinates to perform touch operations; The verification module is used to acquire the current image of the screen under test after the touch terminal of the robotic arm performs the touch operation through the camera, and to perform visual verification on the current image of the screen under test to confirm whether the touch operation produces the expected interface response.

[0007] Thirdly, the present invention also provides a computer device, the computer device including a processor, a network interface, a memory and a communication bus, wherein the processor, the network interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the visual calibration-based touchscreen testing method described in any one of the first aspects above.

[0008] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the touchscreen testing method based on visual calibration as described in any one of the first aspects above.

[0009] Compared with the prior art, the beneficial effects of the present invention are: By using a first region of interest (ROI) and a second ROI, the screen display area within the camera's field of view is independently decoupled and managed hierarchically from the physical working area of ​​the robotic arm. After acquiring the target touch position, it is first mapped to the target pixel coordinates in the camera image. Then, the normalized relative position is calculated by combining the boundary of the first ROI. Finally, the target coordinates of the robotic arm are calculated by combining the boundary of the second ROI. This hierarchical mapping mechanism replaces single-layer direct mapping, avoiding global recalibration when the camera pose is fine-tuned or the device is changed. Furthermore, the normalized ratio calculation effectively absorbs edge positioning errors and slight perspective distortions. After the robotic arm performs a touch operation, the camera is controlled to acquire the current image of the screen under test in real time. The visual algorithm is used to automatically compare and verify the current image to confirm whether the expected interface response has occurred. This constructs a visual feedback closed loop, replacing offline video recording and manual verification, and realizing unattended automated testing. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart of the touchscreen testing method according to an embodiment of the present invention.

[0012] Figure 2 This is a flowchart illustrating the specific steps of step S130 in an embodiment of the present invention.

[0013] Figure 3 This is a flowchart illustrating the specific steps of step S140 in an embodiment of the present invention.

[0014] Figure 4 This is a flowchart illustrating the specific steps of step S142 in an embodiment of the present invention.

[0015] Figure 5 This is a block diagram of a touchscreen testing device according to an embodiment of the present invention.

[0016] Figure 6 This is a schematic block diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0019] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0020] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0021] To address the technical problems in existing robotic arm touchscreen testing, such as the lack of flexibility and large edge errors due to single-layer mapping for coordinate calibration, the absence of visual feedback closed-loop verification, and the inability to automatically confirm interface responses in real time, which severely restricts the realization of unattended automated testing, this invention provides a touchscreen testing method, apparatus, device, and storage medium based on visual calibration.

[0022] The following details the specific steps of a touchscreen testing method based on visual calibration provided by an embodiment of the present invention. This touchscreen testing method is applied to a control terminal, which is connected to a robotic arm and a camera control system. According to the attached... Figure 1-4 As shown, the specific steps of this visual calibration-based touchscreen testing method include: Step S110: Control the camera to capture images of the screen under test, and obtain the first region of interest in the image captured by the camera, and the second region of interest corresponding to the first region of interest when the touch end of the robotic arm touches the display area of ​​the screen under test.

[0023] Specifically, the control terminal controls an industrial camera to take pictures of the vehicle screen under test, acquiring the original image containing the screen display area.

[0024] The first region of interest (ROI) in the image captured by the camera is obtained, i.e., the camera-side ROI. In some embodiments, a target detection model based on the ResNet-50 backbone network can be used to identify the preprocessed image. This target detection model extracts different receptive field features of the image through multi-scale convolutional kernels and combines an attention mechanism to weight and enhance the screen boundary features, thereby outputting the pixel coordinate range of the screen display area in the camera image, including the horizontal start coordinate X1, the horizontal end coordinate X2, the vertical start coordinate Y1, and the vertical end coordinate Y2; for example, the pixel coordinate range of the identified first ROI is (817, 1275) to (1818, 2031).

[0025] Alternatively, testers can also manually drag and drop a rectangle through the interactive interface to select and confirm the first region of interest.

[0026] The tester manipulated the end effector of the robotic arm to move in the horizontal plane using the direction control button, so that it touched the upper left corner and the lower right corner of the screen display area under test respectively. Then, the physical coordinates fed back by the robotic arm controller at this time were recorded, such as (Rx1,Ry1) and (Rx2,Ry2). These two coordinate points together define the boundary of the actual reachable working range of the robotic arm, that is, the second region of interest. Thus, a spatial correspondence was established between the first region of interest and the second region of interest.

[0027] It should be noted that when the camera position changes, only the first region of interest needs to be updated; when the device under test is replaced, only the second region of interest needs to be updated, thus achieving decoupling and hierarchical management of parameters.

[0028] After the robotic arm's touch-sensitive end moves to the target coordinates to perform a touch operation, such as pressing down, holding for 0.4 seconds, and then releasing, the camera re-acquires the current image of the screen under test. Feature extraction and similarity comparison are then performed on this current image to confirm whether the touch operation produced the expected interface response. Feature extraction can be achieved using a convolutional neural network with multi-scale convolutional structures to extract feature vectors from the current image. The comparison logic calculates the cosine similarity between the extracted current image feature vector and the pre-stored expected interface feature vector. If the calculated similarity is greater than a preset similarity threshold, the touch operation is considered to have produced the expected interface response, and the test passes; otherwise, the test fails.

[0029] It should also be noted that, in order to improve the accuracy of subsequent feature extraction and region recognition, the original image needs to be preprocessed. The preprocessing operations include image normalization, cropping and denoising. Specifically, the pixel values ​​of the original image are linearly mapped to the [0,1] interval to achieve normalization, and the image is cropped according to the preset coarse positioning range to reduce computational redundancy. Then, Gaussian filtering algorithm is used for denoising.

[0030] Step S120: Obtain the target touch position of the robotic arm's touch end on the display interface, and map the target touch position to the target pixel coordinates in the image captured by the camera.

[0031] Specifically, to obtain the target touch position of the robotic arm's touch end on the display interface, the control terminal first acquires the current display interface image of the screen under test. To improve the efficiency and accuracy of subsequent model processing, the image data also needs to be preprocessed. Specifically, this preprocessing includes normalization, cropping, and denoising: the pixel values ​​of the original image are linearly scaled to the [0,1] range to achieve normalization; non-display areas are cropped according to preset boundaries to reduce computational redundancy; then, a Gaussian filter operator is used for denoising, with the filter kernel size set to, for example, 3×3. The preprocessed image data will then be transferred to the visual model for feature extraction. Optionally, the size of the preprocessed image is uniformly adjusted to 640×640 pixels to adapt to the input requirements of the subsequent model.

[0032] One approach to obtaining the target touch location is automatic recognition based on a visual model. For example, the visual model uses ResNet-50 as the backbone network structure. The preprocessed image is input into the ResNet-50 model, and the ResNet-50 model extracts deep features from the image through its internal residual block sequence. In order to more accurately capture interface controls (such as buttons, icons, etc.) at different scales, the feature extraction method adopts a multi-scale convolution mechanism, that is, extracting feature maps from network layers at different depths and fusing them. At the same time, an attention mechanism is introduced into the feature extraction network to assign higher weights to regions with significant interactive features in the interface, thereby outputting high-dimensional feature maps rich in semantic information.

[0033] After obtaining the feature map, it is necessary to further determine the target touch location. Specifically, target detection algorithms, such as target detection models based on region convolutional neural networks or the YOLO series, can be used to process the feature map, generate candidate target boxes, and filter the candidate target boxes to calculate the confidence that the candidate box contains the target interface element, such as a navigation button. If the confidence score output by the classifier is greater than a preset threshold, the recognition is considered successful. Subsequently, based on the bounding box coordinates of the recognized target interface element, its geometric center point is calculated. To be compatible with the subsequent coordinate mapping mechanism, the coordinates of this center point are converted into normalized coordinates in the display interface coordinate system, that is, the horizontal and vertical coordinates are proportional values ​​relative to the total width and total height of the display interface, with values ​​ranging from 0 to 1.

[0034] Since the robotic arm's touch position is based on the display interface coordinate system, while subsequent positioning calculations require the camera's field of view coordinate system, a first-level coordinate mapping is necessary. This involves obtaining the normalized coordinates of the target touch position and the pixel boundary range of the first region of interest (ROI) in the camera's view, determined in previous steps. Specifically, this includes the horizontal start coordinate X1, horizontal end coordinate X2, vertical start coordinate Y1, and vertical end coordinate Y2. The target pixel coordinates are calculated using a linear scaling formula: the horizontal coordinate of the target pixel equals the horizontal normalized coordinate multiplied by (X2-X1) plus X1; the vertical coordinate of the target pixel equals the vertical normalized coordinate multiplied by (Y2-Y1) plus Y1. This accurately maps the logical touch point on the display interface to the physical pixel within the camera's field of view. The output target pixel coordinates will be used for subsequent data transfer to calculate the normalized relative position with the boundary of the first ROI.

[0035] For example, the screen resolution is 1920×1080, and the pixel boundary range of the first region of interest captured by the camera is X1=817, X2=1818, Y1=1275, Y2=2031. The control terminal acquires the screen image, performs preprocessing, and then uses the aforementioned target detection algorithm for identification, setting the confidence threshold to 0.95. When the model recognizes the media button, it obtains the normalized coordinates of the center point of its bounding box on the display interface as (0.25, 0.75), which is the target touch position. According to the formula, the horizontal coordinate of the target pixel = 0.25×(1818-817)+817=1067.25; the vertical coordinate of the target pixel = 0.75×(2031-1275)+1275=1842.0. Finally, the floating-point coordinate pair (1067.25, 1842.0) is output, representing the target pixel coordinates of the target touch position in the image captured by the camera.

[0036] Step S130: Calculate the normalized relative position based on the target pixel coordinates and the boundary of the first region of interest.

[0037] Specifically, the control terminal acquires the target pixel coordinates output from the preceding steps, as well as the initial boundary parameters of the first region of interest (ROI) in the camera's captured image, i.e., the display area of ​​the screen under test within the camera's view. After obtaining the precise boundary parameters, it is necessary to verify whether the target pixel coordinates fall within the valid test area. For example, a multi-scale convolution mechanism is used in the feature extraction stage of the visual model, combining feature maps from different receptive fields to capture more comprehensive contextual information. Simultaneously, an attention mechanism is introduced to enhance the weighting of boundary features. Based on the extracted deep features, spatial constraint judgment logic is constructed to determine whether the horizontal value of the target pixel coordinates lies between X1 and X2, and whether the vertical value lies between Y1 and Y2. When the target pixel coordinates are confirmed to be within the first ROI, the calculation of the normalized relative position is performed. Specifically, the validity of the target pixel coordinates is evaluated by determining whether they meet the preset boundary conditions. That is, the coordinates are considered valid when the horizontal value of the target pixel coordinates is between X1 and X2 and the vertical value is between Y1 and Y2. Then, the horizontal relative ratio and the vertical relative ratio are calculated respectively. The horizontal relative ratio is equal to the difference between the horizontal pixel coordinates of the target and the horizontal starting coordinates of the first region of interest, X1, and then divided by the horizontal span of the first region of interest (X2-X1). The vertical relative ratio is equal to the difference between the vertical pixel coordinates of the target and the vertical starting coordinates, Y1, and then divided by the vertical span (Y2-Y1). Both ratio values ​​are between 0 and 1, which is the normalized relative position.

[0038] For example, given a screen resolution of 1920×1080, the target pixel coordinates obtained through previous steps are (1067.25, 1842.0). The precise boundaries of the first region of interest (ROI) obtained through model validation are X1=817, X2=1818, Y1=1275, Y2=2031. During normalization calculations, the horizontal relative proportion is (1067.25-817) / (1818-817)=250.25 / 1001≈0.250; the vertical relative proportion is (1842.0-1275) / (2031-1275)=567.0 / 756=0.750. The final output is a pair of floating-point data (0.250, 0.750), representing the normalized relative position of the target pixel coordinates within the first ROI.

[0039] Step S140: Calculate the target coordinates of the robotic arm based on the normalized relative position and the boundary of the second region of interest.

[0040] Specifically, the second region of interest is the initial parameters of the working range boundary that the robotic arm can reach in the actual physical space. After obtaining the accurate physical boundary parameters, it is necessary to verify whether the normalized relative position can be legally mapped to this physical space. Based on the extracted deep features, a spatial constraint judgment logic is constructed, combined with the normalized relative position, to determine whether it mathematically meets the conditions for triggering physical coordinate calculation, ensuring that the robotic arm's movement will not exceed the boundary due to abnormal floating-point overflow or invalid input. Once the mapping validity is confirmed, the calculation of the robotic arm's physical target coordinates is performed. Specifically, the safety of the mapping can be assessed by verifying whether the value of the normalized relative position belongs to a preset safety range, such as the range [0,1]. When the normalized relative position value is determined to be valid and does not cause any anomalies, the coordinate calculation and subsequent actions are deemed safe and effective. Subsequently, the horizontal coordinate and vertical coordinate of the target of the robotic arm are calculated respectively. The calculation method is as follows: the horizontal coordinate of the target is equal to the horizontal starting value Rx1 of the second region of interest plus the horizontal normalized relative position multiplied by the horizontal span of the second region of interest (Rx2-Rx1); the vertical coordinate of the target is equal to the vertical starting value Ry1 plus the vertical normalized relative position multiplied by the vertical span (Ry2-Ry1).

[0041] For example, the normalized relative position calculated through the previous steps is (0.250, 0.750), and the precise boundary parameters of the second region of interest are Rx1=10, Rx2=105, Ry1=155, Ry2=230; the physical coordinate calculation is performed: the target's horizontal coordinate = 10 + 0.250 × (105 - 10) = 10 + 23.75 = 33.75, and the target's vertical coordinate = 155 + 0.750 × (230 - 155) = 155 + 56.25 = 211.25. The output is a set of floating-point or integer data pairs (33.75, 211.25), which represent the target coordinates of the robotic arm in the physical coordinate system. The target coordinates are then sent.

[0042] Step S150: Control the touch end of the robotic arm to move to the target coordinates to perform a touch operation.

[0043] Specifically, the control terminal sends serial port commands sequentially through a task queue mechanism. First, it sends a command to control the robotic arm's touch terminal to lift off the screen surface. Then, it sends a planar movement command in the format of X and Y coordinates, such as "33.75 211.25", to control the touch terminal to move to the target coordinates. Finally, it sends a pressure command, such as "Z11", indicating a pressure depth of 11 robotic arm coordinate units, so that the touch terminal contacts the screen surface and remains there for a preset time before sending "Z0" to lift it up, thus completing the final drive of the robotic arm to complete the physical touch control.

[0044] Step S160: The camera is used to acquire the current image of the screen under test after the touch operation is performed by the touch end of the robotic arm, and the current image of the screen under test is visually verified to confirm whether the touch operation produces the expected interface response.

[0045] Specifically, deep learning models or traditional image processing algorithms can be used to extract features from the preprocessed image. For example, a convolutional neural network containing a multi-scale convolutional structure can be used to extract high-dimensional semantic features of the image. The network outputs a feature vector of, for example, 2048 dimensions through sliding calculations of multiple convolutional kernels.

[0046] Optionally, in other embodiments, when optical character recognition is used to detect and verify the text response of the interface, the text content in the current screen can be extracted using optical character recognition technology and matched with the expected text string for verification; when image similarity comparison is used, feature extraction focuses on extracting the structural features and color distribution features of the interface; when determining whether the current screen meets the expected interface state, a threshold judgment logic can be used: calculate the cosine similarity between the feature vector of the current screen and the feature vector of the pre-stored expected interface template, and when the similarity is greater than the preset similarity threshold, it is determined that the touch operation has generated the expected interface response.

[0047] In some specific embodiments, the step involves calculating the normalized relative position based on the target pixel coordinates and the boundary of the first region of interest, specifically including the following steps: Step S131: Determine whether the target pixel coordinates are located within the first region of interest.

[0048] Specifically, after the control terminal obtains the target touch position of the robotic arm's touch end on the display interface, it first converts it into target pixel coordinates (x_cam, y_cam) through a first-layer mapping. Then, it compares the target pixel coordinates with the pre-calibrated and stored boundary parameters of the first region of interest (ROI). The boundary parameters of the first ROI include the horizontal start coordinate X1, the horizontal end coordinate X2, the vertical start coordinate Y1, and the vertical end coordinate Y2, where X2 is greater than X1 and Y2 is greater than Y1. The judgment logic is as follows: if the conditions x_cam ≥ X1 and x_cam ≤ X2, and y_cam ≥ Y1 and y_cam ≤ Y2 are met, then the target pixel coordinates are determined to be within the first ROI, and the process proceeds to the next step. If the above conditions are not met, then the target point is determined to be outside the effective display area of ​​the screen being tested, an out-of-bounds alarm message is output, and the current touch operation is terminated or an exception handling mechanism is triggered.

[0049] Step S132: If the target pixel coordinates are located within the first region of interest, calculate the horizontal and vertical relative proportions of the target pixel coordinates based on the horizontal and vertical start and end coordinates of the first region of interest, and use them as the normalized relative position.

[0050] Specifically, once the target pixel coordinates are confirmed to be within the first region of interest (ROI), the relative proportions in the horizontal and vertical directions are calculated separately. Specifically, the horizontal relative proportion *rx* equals the difference between the target's horizontal pixel coordinate *x_cam* and the horizontal starting coordinate *X1* of the ROI, divided by the horizontal span of the ROI, i.e., the difference between *X2* and *X1*. Similarly, the vertical relative proportion *ry* equals the difference between the target's vertical pixel coordinate *y_cam* and the vertical starting coordinate *Y1* of the ROI, divided by the vertical span of the ROI, i.e., the difference between *Y2* and *Y1*. The calculation formulas can be expressed as *rx=(x_cam-X1) / (X2-X1)*; *ry=(y_cam-Y1) / (Y2-Y1)*. The calculated values ​​of *rx* and *ry* are both between 0 and 1, and together they form the two-dimensional normalized relative position (*rx*, *ry*). The normalized relative position, as dimensionless data, will be passed to the third layer of the coordinate mapping layer, that is, the mapping from the normalized relative position to the physical space coordinates of the robotic arm. Combined with the boundary parameters of the second region of interest, the target physical coordinates of the robotic arm will be calculated.

[0051] For example, after the target touch position received by the control terminal is mapped by the first layer, the target pixel coordinates are (1000, 1500); the parameters of the first region of interest that are pre-calibrated are the horizontal start coordinate X1=817, the horizontal end coordinate X2=1818, the vertical start coordinate Y1=1275, and the vertical end coordinate Y2=2031. The system determines whether 1000 is between 817 and 1818, and whether 1500 is between 1275 and 2031. Calculations show that both conditions are met, indicating the target pixel coordinates are within the first region of interest. The horizontal relative proportion rx = (1000-817) / (1818-817) = 183 / 1001 ≈ 0.183, and the vertical relative proportion ry = (1500-1275) / (2031-1275) = 225 / 756 ≈ 0.298. The horizontal span (X2-X1) of the first region of interest is 1001 pixels, and the vertical span (Y2-Y1) is 756 pixels. The final output is a two-dimensional vector of data (0.183, 0.298), representing the normalized relative position.

[0052] In a further embodiment, multiple calibration points are selected within a first region of interest, and the set of pixel coordinates of each calibration point in the image captured by the camera is recorded; the touch end of the robotic arm is moved sequentially to the physical position corresponding to the calibration point, and recorded as the set of physical coordinates of the robotic arm; the step is to calculate the target coordinates of the robotic arm based on the normalized relative position and the boundary of the second region of interest, specifically including the following steps: Step S141: Construct a coordinate mapping relationship based on the pixel coordinate set and the physical coordinate set.

[0053] Specifically, to establish a high-precision coordinate mapping relationship, the control terminal selects N calibration points within the first region of interest (ROI) of the camera's captured image. The calibration points can be selected using the four corner points of the screen or 3×3 or 4×4 grid points (N=9 or 16). For example, when using a 3×3 grid, the system records the pixel coordinates of these 9 calibration points in the camera image, forming a pixel coordinate set P={(x_i,y_i)|i=1,2,...,9}. The tester uses the directional control buttons on the control terminal to move the touch-sensitive end of the robotic arm sequentially to the physical positions on the screen corresponding to the calibration points in step S1. When the end of the robotic arm reaches the physical position of each calibration point, the system reads and records the horizontal and vertical physical coordinates fed back by the robotic arm controller, forming a physical coordinate set Q={(X_i,Y_i)|i=1,2,...,9}. This physical coordinate set corresponds one-to-one with the acquired pixel coordinate set in terms of index, serving as the basic data for subsequently constructing the coordinate mapping relationship. The control terminal associates and stores the acquired pixel coordinate set P with the acquired physical coordinate set Q to form a discrete coordinate lookup table. Based on this lookup table, a coordinate system mapping relationship is constructed. When the calibration points are four corner points, a local coordinate transformation based on quadrilaterals can be used to construct the mapping relationship; when the calibration points are grid points, piecewise linear interpolation or thin-plate spline interpolation methods can be used to construct a global mapping network covering the entire region of interest.

[0054] Step S142: Based on the normalized relative position and coordinate mapping relationship, the target coordinates of the robotic arm are calculated using an interpolation algorithm.

[0055] Specifically, upon receiving the normalized relative position corresponding to the target touch location, the local calibration area into which it falls is first located based on this relative position. Specifically, when using grid calibration points, the target pixel coordinates fall within a local quadrilateral formed by four adjacent calibration points. Let the four vertices of this local quadrilateral be (x_00,y_00), (x_10,y_10), (x_01,y_01), and (x_11,y_11) in the pixel coordinate system, and (X_00,Y_00), (X_10,Y_10), (X_01,Y_01), and (X_11,Y_11) in the robotic arm coordinate system. Then, the normalized local coordinates of the target point within this local quadrilateral are calculated, which is the normalized relative position, where u,v∈[0,1]×[0,1]. Finally, the target coordinates of the robotic arm are calculated using a bilinear interpolation algorithm, with the following formula: X(u,v)=(1-u)(1-v)X_00+u(1-v)X_10+(1-u)vX_01+uvX_11; Y(u,v)=(1-u)(1-v)Y_00+u(1-v)Y_10+(1-u)vY_01+uvY_11.

[0056] The calculated (X(u,v),Y(u,v)) in this step is the final target physical coordinate of the robotic arm, and then a serial port movement command is generated.

[0057] For example, the pixel coordinate set P is the top left corner (817, 1275), top right corner (1818, 1275), bottom left corner (817, 2031), and bottom right corner (1818, 2031); the physical coordinate set Q is the top left corner (10, 155), top right corner (105, 155), bottom left corner (10, 230), and bottom right corner (105, 230); the normalized relative position of the target point is the center of the screen. Four corner points are extracted within the first region of interest as calibration points, and the pixel coordinate set P is recorded. The robotic arm touches the four corner points of the screen sequentially, recording the corresponding physical coordinate set Q. P and Q are associated to construct a quadrilateral-based bilinear interpolation mapping relationship. Since only four corner points are used, the entire screen area is a complete local quadrilateral. The normalized relative position, u=0.5, v=0.5, is received and substituted into the bilinear interpolation formula, as follows: X(0.5,0.5)=(1-0.5)(1-0.5)10+0.5(1-0.5)105+(1-0.5)0.510+0.50.5105=0.2510+0.25105+0.2510+0.25105=57.5; Y(0.5,0.5)=(1-0.5)(1-0.5)155+0.5(1-0.5)155+(1-0.5)0.5230+0.50.5230=0.25155+0.25155+0.25230+0.25230=192.5.

[0058] The optimal values ​​for the key parameters are: number of calibration points N=4; local normalized coordinates u=0.5, v=0.5; and finally, a two-dimensional floating-point vector (57.5, 192.5) is output as the target physical coordinates of the robotic arm.

[0059] In a further embodiment, the step of calculating the target coordinates of the robotic arm using an interpolation algorithm specifically includes the following steps: Step S1421: When the pixel coordinates of the target touch position fall into a local quadrilateral formed by four adjacent calibration points, determine the corresponding vertex coordinates of the local quadrilateral in the pixel coordinate system and the robotic arm coordinate system.

[0060] Specifically, after the control terminal obtains the target pixel coordinates (xcam, ycam) corresponding to the target touch position, it first performs a spatial search within a pre-calibrated multi-calibration point grid (e.g., a 3×3 or 4×4 grid) to determine which four adjacent calibration points the target pixel coordinates fall into within a local quadrilateral region. Once this local quadrilateral is determined, the system extracts the coordinate information of these four vertices from the calibration database. Specifically, these four vertices are defined by their positions as the top-left, top-right, bottom-left, and bottom-right corners, and their coordinates (xcam, ycam) in the pixel coordinate system are obtained respectively. 00 ,y 00 ), (x 10 ,y 10 ), (x 01 ,y 01 ), (x 11 ,y 11 ), and their corresponding target vertex coordinates (X) in the robot arm's physical coordinate system. 00 ,Y 00 ), (X 10 ,Y 10 ), (X 01 ,Y 01 ), (X 11 ,Y 11 ).

[0061] Step S1422: Calculate the normalized local coordinates based on the pixel coordinates of the target touch position and the vertex coordinates of the local quadrilateral in the pixel coordinate system.

[0062] Specifically, since the actual camera used is a low-distortion industrial camera, the captured local screen area can be approximated as a rectangle or a near-parallelogram. The system calculates the normalized local coordinates (u,v) based on the relative position of the target pixel coordinates within the local quadrilateral, where u,v∈[0,1]. The specific calculation method can employ a proportional linear estimation: the horizontal normalized local coordinate u equals the difference between the target horizontal pixel coordinate xcam and the horizontal pixel coordinate of the left boundary, divided by the horizontal pixel span of the local quadrilateral; the vertical normalized local coordinate v equals the difference between the target vertical pixel coordinate ycam and the vertical pixel coordinate of the upper boundary, divided by the vertical pixel span of the local quadrilateral. Optionally, if the camera installation exhibits significant perspective distortion, this step can also employ the method of solving for the inverse of the perspective transformation matrix to accurately calculate the normalized local coordinates (u,v) of the target point within the local quadrilateral.

[0063] Step S1423: Based on the normalized local coordinates and the vertex coordinates of the local quadrilateral in the robot arm coordinate system, the target coordinates of the robot arm are calculated by bilinear interpolation.

[0064] Specifically, the terminal uses the obtained normalized local coordinates and the extracted vertex coordinates of the robotic arm coordinate system to perform bilinear interpolation. The core idea of ​​bilinear interpolation is to first perform two linear interpolations in one direction, and then perform one linear interpolation in the vertical direction. The specific calculation formula is as follows: The horizontal coordinate of the target on the robotic arm is X. robot =(1-u)(1-v)X 00 +u(1-v)X 10 +(1-u)vX 01 +uvX 11 The target longitudinal coordinate of the robotic arm is Y. robot =(1-u)(1-v)Y 00 +u(1-v)Y 10 +(1-u)vY 01 +uvY 11 Using the above formula, the system performs a weighted average of the physical coordinates of the robotic arm at the four vertices of the local quadrilateral, with weights of (u,v), and finally calculates the target coordinates (X) of the robotic arm. robot ,Y robot The target coordinates will be used as input parameters for the execution and control layer to generate serial port movement commands for the robotic arm.

[0065] For example, the target pixel coordinates corresponding to the target touch position are (1200, 1600), and the pixel coordinates of the four adjacent pre-calibrated points are (1000, 1500), (1500, 1500), (1000, 2000), and (1500, 2000) respectively. The physical coordinates of the robotic arm corresponding to the four calibration points are (50, 100) at the top left, (70, 100) at the top right, (50, 120) at the bottom left, and (70, 120) at the bottom right. After retrieval, it is determined that the target pixel coordinates (1200, 1600) fall within the local quadrilateral formed by the above four calibration points. The corresponding vertex coordinates of the quadrilateral in the pixel coordinate system and the robotic arm coordinate system are extracted, and the normalized local coordinates are calculated. For example, the horizontal coordinate u = (1200-1000) / (1500-1000) = 200 / 500 = 0.4; the vertical coordinate v = (1600-1500) / (2000-1500) = 100 / 500 = 0.2.

[0066] Substituting u=0.4 and v=0.2 into the bilinear interpolation formula, the target coordinates of the robotic arm are calculated as follows: Xrobot=(1-0.4)(1-0.2)×50+0.4×(1-0.2)×70+(1-0.4)×0.2×50+0.4×0.2×70; Xrobot=0.48×50+0.32×70+0.12×50+0.08×70=24+22.4+ 6+5.6=58.0; Yrobot=(1-0.4)(1-0.2)×100+0.4×(1-0.2)×100+(1-0.4)×0.2×120+0.4 ×0.2×120; Yrobot=0.48×100+0.32×100+0.12×120+0.08×120=48+32+14.4+9.6=104.0.

[0067] The optimal values ​​for the key parameters are a calibration point grid size of 4×4 and local normalized coordinates u=0.4, v=0.2. Thus, the output two-dimensional floating-point vector (58.0, 104.0) is used as the target physical coordinates of the robotic arm.

[0068] In some specific embodiments, the touchscreen testing method further includes the following steps: Receive touch commands sent by the test script.

[0069] Specifically, the test script orchestration module parses test cases sequentially, generates standard touch commands, and sends them to the touch operation abstraction module of the control terminal. These touch commands include action types, such as click, swipe, long press, and target touch position. The target touch position is mostly represented by normalized coordinates, which are proportional values ​​relative to the width and height of the display interface, with a value range of 0 to 1. These commands do not contain proprietary parameters of the underlying hardware or system interfaces, thus ensuring the universality of the upper-level test scripts.

[0070] Based on the currently selected touch execution mode, the touch commands are routed to either the robotic arm's physical touch mode or the software-injected touch mode.

[0071] Specifically, the touch operation abstraction module maintains an execution mode configuration table to record the currently selected touch execution mode. Upon receiving a touch command, the system reads the mode identifier from this configuration table. If the mode identifier is Mechanical, the touch command is routed to the execution backend of the robotic arm's physical touch mode. If the mode identifier is ADB, the touch command is routed to the execution backend of the software-injected touch mode, thus exposing a unified touch method. This allows upper-layer test scripts to switch or mix and match underlying touch methods without modifying any code logic.

[0072] When routed to the robotic arm physical touch mode, based on the robotic arm coordinate system, the steps are performed to obtain the target touch position of the robotic arm's touch end on the display interface and map the target touch position to the target pixel coordinates in the image captured by the camera.

[0073] Specifically, if the command is routed to the robotic arm's physical touch mode, the system enters a coordinate mapping process based on physical space. First, the received normalized target touch position is converted into the corresponding target pixel coordinates in the camera's captured image. Then, it is determined whether the target pixel coordinates are located within a pre-calibrated first region of interest, and the normalized relative position is calculated. Finally, by combining the boundary parameters of the second region of interest or the interpolation relationship of local multi-calibration points, the target coordinates of the robotic arm in the physical coordinate system are calculated. After the calculation is completed, a pressing depth command is sent to the robotic arm controller to drive the robotic arm end effector to complete the physical touch.

[0074] When routed to the software-injected touch mode, the software touch injection operation is performed based on the pixel coordinate system of the screen being tested.

[0075] Specifically, if the command is routed to the software-injected touch mode, the system enters a system-level virtual touch process. The received normalized target touch position is directly converted into absolute pixel coordinates in the screen pixel coordinate system according to the actual resolution of the screen being tested. For example, the normalized coordinates (0.5, 0.5) are converted into the screen center coordinates (960, 540). Subsequently, virtual touch event commands are sent to the vehicle operating system being tested, such as adb shell inputtap 960 540, and the touch events are directly parsed and generated.

[0076] Therefore, by using the unified touch commands issued by the test script and dynamically routing the commands to either the robotic arm's physical touch mode or the software-injected touch mode based on the currently selected touch execution mode, when the command is routed to the robotic arm's physical touch mode, coordinate mapping and physical action execution are performed based on the robotic arm's coordinate system; when the command is routed to the software-injected touch mode, touch events are directly injected based on the screen pixel coordinate system of the device under test. This process abstracts physical touch and software touch into operations with the same external interface, enabling two touch systems based on different coordinate systems to be seamlessly integrated into the same test framework without the need for complex coordinate synchronization and state handshake mechanisms, thus improving the interchangeability and integration of the test solution.

[0077] In a further embodiment, the touchscreen testing method further includes the following steps: Parse the sequence of test steps in the test script.

[0078] Specifically, the control terminal loads a test script file pre-written by the testers. The script parser reads this file and deserializes the text content into a structured sequence of test steps in memory. This sequence contains multiple sequentially arranged node objects. Each node object contains at least a step type identifier (e.g., Mechanical_Click, Audio_Assert, Vehicle_Info_Check) and the parameter configurations required to execute the step, such as coordinate scale, timeout, and expected assertion values. Additionally, the node objects may also contain logical control instruction identifiers. After parsing, this structured sequence is stored in a task queue, awaiting execution engine invocation.

[0079] According to the sequence of test steps, identify the type of the current test step in turn and execute the corresponding test operation.

[0080] Specifically, the test execution engine retrieves the current test step sequentially from the task queue, identifies its step type identifier, and routes it to the corresponding multimodal processing module. Specifically, when a touch operation is identified, the touch operation abstraction module is invoked to execute physical touch or software injection; when sound acquisition and assertion are identified, the audio acquisition module is invoked to record ambient sound and analyze spectral characteristics; when vehicle remote information acquisition and verification are identified, the on-board diagnostic interface is invoked to obtain vehicle status data. After each test step is completed, the system generates a result object containing the execution status (e.g., pass, fail, exception) and the specific returned data. This result object will serve as the basis for subsequent logic control decisions.

[0081] When a logic control instruction is parsed, the next test step to be executed is determined based on the result of the current test operation or the system state; among which, logic control instructions include conditional branch instructions or loop control instructions.

[0082] Specifically, when the execution engine recognizes the current node as a logical control instruction, the system does not directly execute the specific test action, but instead performs a control flow jump judgment. For conditional branch instructions, it obtains the output result object of the current test operation or queries the system status in real time, such as network connection status and robotic arm running status; if the judgment condition is met, the pointer is pointed to the index of the next step to be executed pointed to by the True branch; if not met, it is pointed to the step index of the False branch.

[0083] It should be noted that for loop control instructions, such as While or For structures, the loop termination condition is evaluated based on the current result or state. If the condition is not met, the pointer is backtracked to the beginning step of the loop body. If the condition is met or the maximum number of loops threshold is reached, the loop is exited and the pointer is pointed to the next sequential step after the loop body. With this setting, the test process is no longer a single linear execution, but can be dynamically and adaptively adjusted according to the actual feedback from the device under test.

[0084] In some specific embodiments, the step involves visually verifying the current image of the screen under test to confirm whether the touch operation produces the expected interface response, specifically including the following steps: Determine the expected response area corresponding to the touch operation and obtain the dynamic ignore area mask of the screen under test, wherein the dynamic ignore area mask at least covers the status bar display area of ​​the screen under test.

[0085] Specifically, after the robotic arm or software injection completes a touch operation, the control terminal determines the expected response area corresponding to the touch operation based on the pre-configured interface element position information in the test script. This area is represented in the form of pixel coordinate range. Meanwhile, since the status bar at the top of the vehicle screen usually displays dynamically changing content such as real-time time, network signal, and Bluetooth status, direct full-image comparison is prone to false alarms. Therefore, by obtaining a pre-generated or dynamically updated dynamic ignore area mask, which is mostly a binary image matrix consistent with the screen resolution, at least covering the status bar display area of ​​the screen under test, the pixel value corresponding to the status bar in the mask is set to the first logical value (0 indicates ignore), and the remaining areas are set to the second logical value (1 indicates reserve).

[0086] Acquire a preset interface image, and extract the target comparison area between the current screen of the test screen and the preset interface image based on the expected response area and the dynamic ignore area mask.

[0087] Specifically, the control terminal retrieves the preset interface image corresponding to the current test step from the storage medium, i.e., the standard template image, and obtains the current image of the screen under test captured by the camera. It performs isomorphic processing on the current image and the preset interface image. For example, firstly, according to the coordinate range of the determined expected response area, the two images are pre-processed by cropping to remove irrelevant background areas. Then, the dynamic ignored area mask is logically ANDed with the cropped current image and the preset interface image to set the pixel areas marked as ignored in the mask to zero or remove them.

[0088] After the above cropping and masking filtering, the target comparison region containing only the expected interface response features and excluding dynamic interference is extracted from the two images.

[0089] Calculate the structural similarity of the target comparison region; if the structural similarity is greater than the preset similarity threshold, then confirm that the touch operation produces the expected interface response.

[0090] Specifically, the visual verification module uses a structural similarity algorithm to measure the similarity between the two extracted target comparison regions. Unlike the traditional mean square error, which only focuses on pixel-level differences, SSIM comprehensively evaluates image similarity from three dimensions: brightness, contrast, and structure. The specific calculation process includes calculating the local mean, local variance, and covariance between the two target comparison regions respectively, and substituting them into the SSIM calculation formula. The formula involves adjusting the hyperparameters of brightness, contrast, and structure weights. For example, the default value is 1, and the calculation result is a floating-point value between -1 and 1. The closer the value is to 1, the higher the structural similarity between the two images.

[0091] This system has a preset similarity threshold for determining whether the verification passes. This preset similarity threshold can be configured according to different test scenarios or interface complexity. For example, in some embodiments with high requirements for interface consistency, the threshold can be set to a value between 0.85 and 0.95. The calculated structural similarity value is compared with the preset threshold: if the structural similarity is greater than the preset similarity threshold, it is determined that the interface state of the current screen under test is consistent with the expected state, confirming that the touch operation has produced the expected interface response, and the verification pass result is output; if the structural similarity is less than or equal to the preset threshold, it is determined that the expected response has not been produced, and the verification fail result is output.

[0092] For example, the current screen being tested is a 1920×1080 resolution image captured by a camera, with the top 0 to 60 pixels forming the status bar, and the current display time is 14:30; the preset interface image is a pre-stored standard template of the air conditioner settings main interface, with the status bar time being 09:00; the expected response area coordinates are (200,200) to (1500,900); the preset similarity threshold is 0.90. The test script configuration is read, determining the expected response area to be (200,200) to (1500,900); a pre-generated binarized dynamically ignored region mask is obtained, where the value in the Y-axis row from 0 to 60 pixels is 0, and the value in the remaining areas is 1. Local images with coordinates ranging from (200,200) to (1500,900) are cropped from both the current screen and the preset interface image; then, the dynamically ignored region mask is applied to filter out any remaining status bar interference pixels in the local images, resulting in... Two target comparison regions containing only features of the air conditioning settings interface were selected. SSIM calculation was performed on the two target comparison regions. By solving for the local mean, variance, and covariance, the structural similarity score was obtained as 0.96. Then, the score of 0.96 was compared with the preset threshold of 0.90. It was determined that 0.96 > 0.90, and the expected response region clipping boundary was (200, 200, 1500, 900). The SSIM similarity threshold was 0.90. Therefore, the verification result passed, and the SSIM score of 0.96 was recorded in the test log.

[0093] Therefore, by dynamically ignoring the region mask, interference caused by high-frequency dynamic changes in areas such as time and signal in the vehicle screen status bar is effectively eliminated, avoiding false alarms in overall verification due to local dynamic pixel differences. This improves the accuracy of response confirmation in complex interface states, while also increasing computational efficiency and reducing resource consumption. The entire image is first pre-processed by cropping through the expected response area, which significantly reduces the dimension of the image matrix involved in the calculation. Subsequently, SSIM calculation is performed only within the reduced target comparison area, avoiding full image traversal and solving, thereby reducing the CPU and memory computation overhead of the control terminal and ensuring real-time response of visual verification.

[0094] The present invention also provides a touch screen testing device 100 based on visual calibration. The touch screen testing device 100 is disposed on a control terminal, which is connected to a robotic arm and a camera control. The touch screen testing device 100 based on visual calibration is used to execute any embodiment of the aforementioned touch screen testing method based on visual calibration.

[0095] Specifically, the following describes in detail the specific structure of a touchscreen testing device 100 based on visual calibration provided by an embodiment of the present invention, according to the appendix. Figure 5 As shown, the specific structure of the visual calibration-based touchscreen testing device 100 includes the following: The acquisition module 110 is used to control the camera to capture the screen under test, and to acquire a first region of interest in the image captured by the camera, and a second region of interest corresponding to the first region of interest when the touch end of the robotic arm touches the display area of ​​the screen under test. The first mapping module 120 is used to obtain the target touch position of the touch end of the robotic arm on the display interface, and map the target touch position to the target pixel coordinates in the image captured by the camera; Calculation module 130 is used to calculate the normalized relative position based on the target pixel coordinates and the boundary of the first region of interest; The second mapping module 140 is used to calculate the target coordinates of the robotic arm based on the normalized relative position and the boundary of the second region of interest; The control module 150 is used to control the touch end of the robotic arm to move to the target coordinates to perform a touch operation; The verification module 160 is used to acquire the current image of the screen under test after the touch terminal of the robotic arm performs the touch operation through the camera, and to perform visual verification on the current image of the screen under test to confirm whether the touch operation produces the expected interface response.

[0096] The present invention further provides a computer device, which can be used to perform the visual calibration-based touch screen testing method in any of the above embodiments.

[0097] See Figure 6 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a communication bus 501. The memory may include a storage medium 503 and internal memory 504.

[0098] The storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it causes the processor 502 to execute a touch screen testing method based on visual calibration. The storage medium 503 may be a volatile storage medium or a non-volatile storage medium.

[0099] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0100] The internal memory 504 provides an environment for the operation of the computer program 5032 in the storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a touch screen testing method based on visual calibration.

[0101] This network interface 505 is used for network communication, such as providing data transmission. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device 500 to which the present invention is applied. The specific computer device 500 may include, but is not limited to, the following: Figure 6 The diagram shows more or fewer components, or combinations of certain components, or different component arrangements.

[0102] The processor 502 is used to run the computer program 5032 stored in the memory to implement the corresponding functions in the above-mentioned touch screen testing method based on visual calibration.

[0103] Those skilled in the art will understand that Figure 6 The embodiments of the computer device shown do not constitute a limitation on the specific configuration of the computer device. In other embodiments, the computer device may include, but is not limited to, other components. Figure 6 More or fewer components, or combinations of certain components, or different arrangements of components. For example, in some embodiments, a computer device may include only memory and a processor; in such embodiments, the structure and function of the memory and processor are different from those of other components. Figure 6 The embodiments shown are consistent and will not be repeated here.

[0104] It should be understood that, in this embodiment of the invention, the processor 502 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0105] In another embodiment of the invention, a computer-readable storage medium is provided. This computer-readable storage medium may be volatile or non-volatile. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps included in the above-described visual calibration-based touchscreen testing method.

[0106] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0107] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Units with the same function may be grouped into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, or it may be an electrical, mechanical, or other form of connection.

[0108] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0109] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0110] The integrated units described above can be implemented in either hardware or software functional units.

[0111] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, a data transmitter, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned computer-readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks.

[0112] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A touchscreen testing method based on visual calibration, characterized in that, The touchscreen testing method is applied to a control terminal, which is connected to a robotic arm and a camera control system. The touchscreen testing method includes: The camera is controlled to capture images of the screen under test, and a first region of interest is obtained from the image captured by the camera, and a second region of interest corresponding to the first region of interest is obtained when the touch end of the robotic arm touches the display area of ​​the screen under test. The target touch position of the robotic arm's touch end on the display interface is obtained, and the target touch position is mapped to the target pixel coordinates in the image captured by the camera; Calculate the normalized relative position based on the target pixel coordinates and the boundary of the first region of interest; The target coordinates of the robotic arm are calculated based on the normalized relative position and the boundary of the second region of interest; Control the touch end of the robotic arm to move to the target coordinates to perform a touch operation; The camera captures the current image of the screen under test after the touch operation is performed by the touch terminal of the robotic arm, and performs visual verification on the current image of the screen under test to confirm whether the touch operation produces the expected interface response.

2. The touchscreen testing method based on visual calibration according to claim 1, characterized in that, The step of calculating the normalized relative position based on the target pixel coordinates and the boundary of the first region of interest includes: Determine whether the target pixel coordinates are located within the first region of interest; If the target pixel coordinates are located within the first region of interest, then the horizontal and vertical relative proportions of the target pixel coordinates are calculated based on the horizontal and vertical start and end coordinates of the first region of interest, and used as the normalized relative position.

3. The touchscreen testing method based on visual calibration according to any one of claims 1 or 2, characterized in that, Multiple calibration points are selected within the first region of interest, and the set of pixel coordinates of each calibration point in the image captured by the camera is recorded; the touch end of the robotic arm is moved sequentially to the physical position corresponding to the calibration point, and recorded as the set of physical coordinates of the robotic arm; The step of calculating the target coordinates of the robotic arm based on the normalized relative position and the boundary of the second region of interest includes: A coordinate mapping relationship is constructed based on the pixel coordinate set and the physical coordinate set; Based on the normalized relative position and the coordinate mapping relationship, the target coordinates of the robotic arm are calculated using an interpolation algorithm.

4. The touchscreen testing method based on visual calibration according to claim 3, characterized in that, The calculation of the target coordinates of the robotic arm using an interpolation algorithm includes: When the pixel coordinates of the target touch position fall into a local quadrilateral formed by four adjacent calibration points, the corresponding vertex coordinates of the local quadrilateral in the pixel coordinate system and the robotic arm coordinate system are determined. Calculate the normalized local coordinates based on the pixel coordinates of the target touch position and the vertex coordinates of the local quadrilateral in the pixel coordinate system; Based on the normalized local coordinates and the vertex coordinates of the local quadrilateral in the robot arm coordinate system, the target coordinates of the robot arm are calculated by bilinear interpolation.

5. The touchscreen testing method based on visual calibration according to claim 1, characterized in that, The touchscreen testing method also includes: Receive touch commands sent by the test script; Based on the currently selected touch execution mode, the touch command is routed to either the robotic arm's physical touch mode or the software-injected touch mode; When the route is to the physical touch mode of the robotic arm, based on the robotic arm coordinate system, the steps of obtaining the target touch position of the touch end of the robotic arm on the display interface and mapping the target touch position to the target pixel coordinates in the image captured by the camera are performed. When routed to the software-injected touch mode, the software touch injection operation is performed based on the pixel coordinate system of the screen being tested.

6. The touchscreen testing method based on visual calibration according to claim 5, characterized in that, The touchscreen testing method also includes: Parse the test step sequence in the test script; According to the arrangement order of the test step sequence, the type of the current test step is identified in turn, and the corresponding test operation is executed. When a logic control instruction is parsed, the next test step to be executed is determined based on the result of the current test operation or the system state; wherein, the logic control instruction includes conditional branch instruction or loop control instruction.

7. The touchscreen testing method based on visual calibration according to claim 1, characterized in that, The step of visually verifying the current screen display to confirm whether the touch operation produces the expected interface response includes: Determine the expected response area corresponding to the touch operation and obtain the dynamic ignore area mask of the screen under test, wherein the dynamic ignore area mask at least covers the status bar display area of ​​the screen under test; Acquire a preset interface image, and extract the target comparison region between the current screen of the tested screen and the preset interface image based on the expected response region and the dynamic ignore region mask; Calculate the structural similarity of the target comparison region; If the structural similarity is greater than a preset similarity threshold, then the touch operation is confirmed to produce the expected interface response.

8. A touchscreen testing device based on visual calibration, characterized in that, The touchscreen testing device is located in a control terminal, which is connected to the robotic arm and camera control. The touchscreen testing device includes: The acquisition module is used to control the camera to capture images of the screen under test, and to acquire a first region of interest in the image captured by the camera, and a second region of interest corresponding to the first region of interest when the touch end of the robotic arm touches the display area of ​​the screen under test. The first mapping module is used to obtain the target touch position of the touch end of the robotic arm on the display interface, and map the target touch position to the target pixel coordinates in the image captured by the camera; The calculation module is used to calculate the normalized relative position based on the target pixel coordinates and the boundary of the first region of interest; The second mapping module is used to calculate the target coordinates of the robotic arm based on the normalized relative position and the boundary of the second region of interest; The control module is used to control the touch end of the robotic arm to move to the target coordinates to perform touch operations; The verification module is used to acquire the current image of the screen under test after the touch terminal of the robotic arm performs the touch operation through the camera, and to perform visual verification on the current image of the screen under test to confirm whether the touch operation produces the expected interface response.

9. A computer device, characterized in that, The computer device includes a processor, a network interface, a memory, and a communication bus, wherein the processor, network interface, and memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the touchscreen testing method based on visual calibration as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the touch screen testing method based on visual calibration as described in any one of claims 1-7.

Images