Method, device and equipment for automatic testing of capacitive sensing products and storage medium

Abstract

The application discloses an automatic testing method, device and equipment for a capacitive sensing product and a storage medium. The method comprises the following steps: acquiring a global image of a product to be tested, determining an image coordinate corresponding to a target feature point in the global image in a preset image coordinate system; controlling a robot arm to move to a first physical position corresponding to the target feature point, and determining a motion coordinate of the target feature point in a robot arm motion coordinate system; determining a mapping matrix from the preset image coordinate system to the robot arm motion coordinate system according to the image coordinate and the motion coordinate; acquiring a first coordinate of a touch point to be tested in the preset image coordinate system, mapping the first coordinate to a second coordinate of the touch point to be tested in the robot arm motion coordinate system according to the mapping matrix; and controlling the robot arm to perform a touch operation on the product to be tested according to the second coordinate. The automatic testing method for the capacitive sensing product can significantly improve the testing efficiency and the testing quality.

Description

Technical Field

[0001] This application relates to the field of capacitive sensing product testing technology, and in particular to an automated testing method, apparatus, equipment and storage medium for capacitive sensing products. Background Technology

[0002] With the widespread application of projected capacitive sensing technology in products such as smartphones, tablets, automotive touchscreens, smart wearable devices, and capacitive pens, the performance indicators of capacitive sensing products, such as touch accuracy, response speed, trajectory linearity, and multi-touch synchronization, have become core elements in measuring product interaction experience and overall quality. Therefore, comprehensive and accurate testing of the aforementioned touch performance is crucial in the product development, verification, factory quality inspection, and reliability testing stages.

[0003] In related technologies, the touch performance testing of capacitive sensing products mainly relies on manual operation. However, significant differences exist in the touch pressure, movement speed, pressing angle, and touch position among different testers, resulting in large dispersion and poor repeatability of multiple test results for the same product, making it impossible to establish a unified testing standard. As the performance requirements of touch products continue to increase, traditional manual testing methods are no longer sufficient to meet the needs of high-precision and standardized testing. Summary of the Invention

[0004] This application provides an automated testing method, apparatus, equipment, and storage medium for capacitive sensing products to solve the technical problem that traditional manual testing methods can hardly meet the requirements for high-precision and standardized testing.

[0005] To address the aforementioned technical problems, in a first aspect, this application provides an automated testing method for capacitive sensing products, comprising: acquiring a global image of the product under test, determining the image coordinates of a target feature point in the global image under a preset image coordinate system; controlling a robotic arm to move to a first physical position corresponding to the target feature point, and determining the motion coordinates of the target feature point under the robotic arm motion coordinate system; Based on the image coordinates and the motion coordinates, a mapping matrix is ​​determined from the preset image coordinate system to the robotic arm motion coordinate system; the first coordinates of the touch point to be tested in the preset image coordinate system are obtained, and the first coordinates are mapped to the second coordinates of the touch point to be tested in the robotic arm motion coordinate system according to the mapping matrix; the robotic arm is controlled to perform touch operation on the product under test according to the second coordinates.

[0006] In this automated testing method for capacitive sensing products, a full-area image of the product under test is acquired, and the image coordinates of target feature points are extracted. This digitizes the actual placement position of the product, providing a data foundation for coordinate mapping. By controlling a robotic arm to touch the feature points and determining their motion coordinates in the robotic arm's own coordinate system, the spatial position corresponding to the image is reproduced in the robotic arm's own coordinate system. The mapping matrix between the two coordinate systems is determined using the correspondence between image coordinates and motion coordinates. The image coordinates of the touch point under test are then substituted into the mapping matrix to convert them into motion coordinates, ensuring an accurate correspondence between the theoretical test point and the actual action. Controlling the robotic arm to perform touch operations based on the motion coordinates eliminates touch position deviations caused by product placement offsets or robotic arm positioning errors, ensuring the accuracy and repeatability of the test results.

[0007] In some embodiments, determining the image coordinates of the target feature points in the global image under a preset image coordinate system includes: performing distortion correction and edge contour extraction on the global image to identify the effective touch area; within the effective touch area, determining target feature points that meet preset feature recognition conditions, including corner response conditions and edge gradient direction change conditions; and performing sub-pixel-level position fitting on the target feature points based on the grayscale distribution of the target feature points in the local area to obtain the image coordinates of the target feature points under the preset image coordinate system.

[0008] In this embodiment, distortion correction eliminates lens optical errors, and edge contour extraction segments the effective touch area to eliminate background interference. Target feature points are screened using both corner response and edge gradient direction changes to ensure stable recognition under different lighting and placement angles. Finally, sub-pixel-level position fitting improves the feature point positioning accuracy from pixel-level to sub-pixel-level, providing high-precision input data for accurate solution of the mapping matrix, thereby ensuring the positioning accuracy of the entire testing system.

[0009] In some embodiments, controlling the robotic arm to move to the first physical position corresponding to the target feature point and determining the motion coordinates of the target feature point in the robotic arm's motion coordinate system includes: controlling the robotic arm to move to the first physical position corresponding to the target feature point, causing the robotic arm's touch head to contact the touch surface at the first physical position; reading the encoders of each joint of the robotic arm to obtain multiple joint encoding values; and performing kinematic calculations on the multiple joint encoding values ​​according to the forward kinematic model of the robotic arm to obtain the motion coordinates of the target feature point in the robotic arm's motion coordinate system.

[0010] In this embodiment, multiple joint angle values ​​are obtained by reading the encoders of each joint of the robotic arm, and kinematic calculations are performed using a forward kinematic model to obtain the three-dimensional spatial coordinates of the end effector. This method does not rely on external measuring equipment and is unaffected by environmental factors such as lighting and occlusion, ensuring the stability and reliability of the motion coordinate data. Compared with visual positioning, kinematic calculation has a higher data update frequency and lower latency, enabling real-time feedback of the precise position of the robotic arm's end effector.

[0011] In some embodiments, determining the mapping matrix from the preset image coordinate system to the robotic arm motion coordinate system based on the image coordinates and the motion coordinates includes: generating a first coordinate point set and a second coordinate point set, wherein the first coordinate point set includes the image coordinates of multiple target feature points, and the second coordinate point set includes the motion coordinates of multiple target feature points; constructing an overdetermined system of equations based on the first coordinate point set, the second coordinate point set, and the linear transformation relationship from the image coordinate system to the robotic arm motion coordinate system, wherein the linear transformation relationship includes rotation matrix components and translation vector components; and solving the overdetermined system of equations using the least squares method to obtain the mapping matrix from the preset image coordinate system to the robotic arm motion coordinate system.

[0012] In this embodiment, a first coordinate point set and a second coordinate point set are constructed using the image coordinates and motion coordinates of multiple sets of feature points. The mapping matrix is ​​solved using an overdetermined system of equations and the least squares method, minimizing the overall fitting error of all corresponding point pairs and effectively suppressing the impact of single-point measurement noise on mapping accuracy. The redundant information provided by multiple feature points makes the solution of the mapping matrix statistically optimal. Even if there are slight deviations in the coordinate measurements of individual feature points, it will not significantly affect the final mapping accuracy, thus solving the problem of coordinate mapping deviation caused by product placement offset.

[0013] In some embodiments, controlling the robotic arm to perform touch operations on the product under test according to the second coordinates includes: determining basic gestures corresponding to the touch operations based on the testing requirements of the product under test, wherein the basic gestures include at least one of single click, double click, long press, straight swipe, curved swipe, edge touch, multi-finger zoom, multi-finger rotation, and fine writing; determining gesture parameters corresponding to each of the basic gestures, wherein the gesture parameters include spatial motion trajectory, motion speed, and touch pressure; generating a trajectory command stream based on the execution order of the basic gestures and the gesture parameters; and controlling the robotic arm to perform touch operations on the product under test according to the second coordinates and the trajectory command stream.

[0014] In this embodiment, basic gestures are selected from a preset gesture library, and parameters such as the spatial motion trajectory, motion speed, and touch pressure corresponding to each gesture are set and combined sequentially according to the execution order to automatically generate a trajectory instruction stream that can be executed by the robotic arm. This transforms diverse testing requirements into specific actions that the robotic arm can perform, achieving flexibility in test scheme configuration and a high degree of automation in robotic arm execution. It can adapt to different types of touch product testing without manual programming, significantly improving the versatility and ease of use of the testing system.

[0015] In some embodiments, controlling the robotic arm to perform touch operations on the tested product according to the second coordinate and the trajectory command flow includes: controlling the robotic arm to move to the second physical position corresponding to the second coordinate, so that the robotic arm touch head contacts the touch surface at the second physical position; collecting the touch pressure value of the robotic arm touch head at a preset sampling frequency, and determining the pressure deviation value between the touch pressure value and the preset target pressure value; determining the height adjustment amount and posture adjustment amount of the robotic arm touch head according to the pressure deviation value, and adjusting the height and posture of the robotic arm touch head according to the height adjustment amount and the posture adjustment amount, respectively.

[0016] In this embodiment, the touch pressure value is acquired in real time through high-frequency sampling, the deviation from the target pressure is calculated, and the height and posture adjustment amounts are determined based on the pressure deviation. This drives the robotic arm to perform displacement and angle corrections, forming a closed-loop feedback of acquisition, calculation, adjustment, and re-acquisition. Even if there are minor deformations on the product surface or vibrations in the robotic arm, the system can respond in real time and dynamically correct the end position and posture, ultimately ensuring that the touch pressure stabilizes and converges to the target value, guaranteeing the consistency of pressure throughout the entire touch process.

[0017] In some embodiments, after adjusting the height and posture of the robotic arm's touch head according to the height adjustment amount and the posture adjustment amount, the method further includes: acquiring the touch coordinate sequence and touch time sequence reported by the product under test; determining the touch accuracy and trajectory linearity of the current touch operation based on the touch coordinate sequence; determining the multi-point synchronization error and response delay time of the current touch operation based on the touch time sequence; and determining the current test result based on the touch accuracy, the trajectory linearity, the multi-point synchronization error, and the response delay time.

[0018] In this embodiment, the touch coordinate sequence and touch time sequence reported by the product under test are acquired synchronously. The touch accuracy and trajectory linearity are determined based on the coordinate sequence, and the multi-point synchronization error and response delay time are determined based on the time sequence. This upgrades the traditional subjective qualitative evaluation to an objective quantitative evaluation. The four performance indicators jointly determine the current test result, and the test result is traceable, allowing for precise identification of the causes of product defects.

[0019] Secondly, this application provides an automated testing device for capacitive sensing products, comprising: The first determining module is used to acquire a full-domain image of the product under test and determine the image coordinates of the target feature points in the full-domain image in a preset image coordinate system. The second determining module is used to control the robotic arm to move to the first physical position corresponding to the target feature point, and to determine the motion coordinates of the target feature point in the robotic arm motion coordinate system. The third determining module is used to determine a mapping matrix from the preset image coordinate system to the robotic arm motion coordinate system based on the image coordinates and the motion coordinates. The mapping module is used to obtain the first coordinates of the touch point to be tested in the preset image coordinate system, and to map the first coordinates to the second coordinates of the touch point to be tested in the robot arm motion coordinate system according to the mapping matrix; The touch module is used to control the robotic arm to perform touch operations on the product under test according to the second coordinate.

[0020] Thirdly, this application provides a computer device, characterized in that it includes a processor and a memory, the memory being used to store a computer program, which, when executed by the processor, implements the automated testing method for capacitive sensing products as described above.

[0021] Fourthly, this application provides a computer-readable storage medium, characterized in that it stores a computer program, which, when executed by a processor, implements the above-described automated testing method for capacitive sensing products. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating an automated testing method for capacitive sensing products according to an embodiment of this application. Figure 2 This is a schematic diagram of the structure of an automated testing device for capacitive sensing products shown in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0024] Please refer to Figure 1 , Figure 1This is a flowchart illustrating the automated testing method for capacitive sensing products provided in an embodiment of this application. Figure 1 As shown, the automated testing method for capacitive sensing products in this embodiment includes steps S11 to S15, which are detailed below: Step S11: Obtain the global image of the product under test, and determine the image coordinates of the target feature points in the global image in the preset image coordinate system.

[0025] In this step, a full-area image of the touch area of ​​the product under test is acquired through a visual positioning unit, and clearly identifiable feature points are extracted as target feature points, such as the four corners of the screen, edge intersections, and specific marker points. The image coordinates of the target feature points can be determined using the preset image coordinate system of the visual positioning unit. The actual spatial position information of the product under test is digitized into image coordinate data, providing the initial data foundation for establishing the coordinate mapping relationship between the product under test and the robotic arm, while implicitly capturing the actual placement information of the product under test.

[0026] In some embodiments, when performing step S11, firstly, distortion correction and edge contour extraction are performed on the global image of the product under test to identify the effective touch area; then, within the effective touch area, target feature points that meet preset feature recognition conditions are determined, including corner response conditions and edge gradient direction change conditions; based on the grayscale distribution of the target feature points in the local area, sub-pixel-level position fitting is performed on the target feature points to obtain the image coordinates corresponding to the target feature points in the preset image coordinate system.

[0027] In this step, the industrial camera lens of the visual positioning unit exhibits radial and tangential distortion, causing non-linear shifts in pixel positions at image edges. Distortion correction eliminates coordinate errors caused by lens distortion by performing an inverse transformation on the entire image using pre-calibrated camera intrinsic parameters. Edge contour extraction utilizes gradient detection operators, such as the Canny operator, to identify the boundaries of the touch area of ​​the tested product, thereby segmenting the effective touch area from the background image and avoiding the influence of non-touch areas such as borders, reflections, and environmental interference on subsequent feature detection.

[0028] A corner point is a point in an image where the gray level changes significantly in two orthogonal directions, such as the four corners of a screen or the corners of a rectangular marker. The corner response condition is as follows: calculate the corner response value of each pixel in the image; if the corner response value is greater than a preset threshold, it is considered to meet the corner response condition. To determine the corner response value, calculate the product and sum of the gray level changes of the target pixel in the two orthogonal directions, and determine the corner response value as the ratio of the product to the sum. To determine the corner response value, first calculate the degree of gray level change of each pixel in the horizontal and vertical directions, and then construct the corner response value based on these gray level changes. When the corner response value of a pixel is greater than the preset corner response threshold, that point is determined as a candidate corner point. An edge is a region in an image where the gray level changes abruptly, and a corner point can be considered the intersection of two edges. At edges, the gradient direction is relatively consistent; however, at corner points, the gradient direction changes significantly. For example, a horizontal edge becomes a vertical direction at a corner point. By analyzing the gradient direction distribution within the neighborhood of a candidate point, it is determined whether the point is a corner or intersection. Specifically, the gradient direction of each pixel within the neighborhood of the candidate point is statistically analyzed, and the degree of change in the gradient direction is calculated. When the change in gradient direction exceeds a preset threshold, the point is determined to meet the edge gradient direction change condition. Through the comprehensive screening of the corner response condition and the edge gradient direction change condition, the finally identified target feature points are located within the effective touch area and have a clear geometric position. They also exhibit good stability and repeatability, and can be reliably detected under different lighting conditions and different placement angles, thus ensuring the consistency and reliability of the mapping matrix.

[0029] Conventional pixel-level corner detection has limited positioning accuracy due to the discretization of image pixels, achieving only ±1 pixel precision, which cannot meet the requirements of high-precision touch testing. Sub-pixel-level position fitting, based on pixel-level detection results, utilizes the grayscale distribution information of the local region where the target feature point is located to interpolate and refine the precise position of the feature point, improving positioning accuracy to the sub-pixel level. When performing sub-pixel-level position fitting, the pixel position of the target feature point is used as the reference point. Centered on the surrounding area The gray-level matrix within the neighborhood. A quadratic surface fitting method is used to approximate the gray-level distribution within the neighborhood as a quadratic polynomial surface: ;in, These are the offset coordinates of the neighboring pixels relative to the center pixel. Let a, b, c, d, e, and f be the grayscale values ​​of the corresponding pixels, and a, b, c, d, e, and f be the surface parameters to be fitted. After solving for these six parameters using the least squares method with respect to the grayscale values ​​of the neighboring pixels, the extreme points of this quadratic surface are the sub-pixel level corner positions. Specifically, by taking the partial derivatives with respect to x and y and setting them to zero, the sub-pixel offset can be obtained. Then the sub-pixel coordinates of the target feature point in the image coordinate system are: ; ;in, These are the corner coordinates detected at the pixel level. This is the sub-pixel offset, which is the positional offset of the extreme point of the quadratic surface relative to the center pixel. The above process fully utilizes the gray-level distribution information in the neighborhood of the target feature point, achieving fine-grained correction of the feature point position. This makes the feature point positioning accuracy far higher than that of conventional pixel-level positioning methods, ensuring the accuracy of subsequent mapping matrix solving.

[0030] Step S12: Control the robotic arm to move to the first physical position corresponding to the target feature point, and determine the motion coordinates of the target feature point in the robotic arm motion coordinate system.

[0031] In this step, the end effector of the robotic arm sequentially touches the actual physical position corresponding to each target feature point, and calculates the three-dimensional coordinates of each feature point in the robotic arm's motion coordinate system by reading the values ​​of the robotic arm joint encoder and combining them with the forward kinematics model. The spatial position corresponding to the image coordinates is reproduced in the robotic arm's own motion coordinate system.

[0032] In some embodiments, when performing step S12, the robotic arm is first controlled to move to the first physical position corresponding to the target feature point, so that the robotic arm touch head contacts the touch surface at the first physical position; the encoders of each joint of the robotic arm are read to obtain multiple joint encoding values; and kinematic calculations are performed on the multiple joint encoding values ​​according to the forward kinematic model of the robotic arm to obtain the motion coordinates of the target feature point in the robotic arm motion coordinate system.

[0033] In this step, within the kinematics of the robotic arm, forward kinematics refers to determining the position of the end effector in the reference coordinate system based on the joint angles of each joint. The encoder feedback values ​​of each joint, i.e., the encoded values ​​of each joint, are used as known inputs. The forward kinematics model is then used to calculate the precise three-dimensional position coordinates of the touch head in the robotic arm's motion coordinate system.

[0034] Taking a six-DOF serial robotic arm as an example, a fixed coordinate system is first established for each joint of the robotic arm. Using the Denavit-Hartenberg parametric method, a joint coordinate system is established at each joint. The spatial transformation relationship between any two adjacent joint coordinate systems can be described by four geometric parameters: the link length and link twist angle between the two joint axes are determined by the physical structure of the robotic arm and are fixed values; the joint offset along the joint axis and the joint angle around the joint axis, where the joint angle is directly read by the encoder. For a serial robotic arm with n joints, the transformation from the base coordinate system to the coordinate system of the end effector requires n successive transformations between adjacent coordinate systems. Each transformation is completed through a 4×4 homogeneous transformation matrix, which simultaneously includes rotation and translation operations, transforming the coordinates in the previous joint coordinate system to the current joint coordinate system.

[0035] In the specific calculation, the joint angle values ​​read by the encoder are sequentially substituted into the homogeneous transformation matrices of the coordinate systems of adjacent joints, from the base to the end effector. These matrices are then multiplied sequentially to obtain a total transformation matrix. The upper left 3×3 portion of this total transformation matrix describes the spatial orientation of the end effector, while the upper right 3×1 portion represents the three-dimensional position coordinates of the end effector in the base coordinate system. Extracting these three-dimensional position coordinates yields the motion coordinates of the target feature point in the robotic arm's motion coordinate system. Substituting the rotation angles of each joint into a pre-established sequence of link transformation matrices, the total pose of the end effector relative to the base is obtained through matrix multiplication. The core of this method lies in utilizing the encoder readings of the robotic arm itself, enabling precise determination of the spatial position of the end effector without external measuring equipment. By combining the joint encoder with the forward kinematics model, precise quantification of the end effector position is achieved. This method does not rely on external measuring equipment and can acquire high-precision motion coordinate data without additional vision or laser positioning systems. Furthermore, this method is unaffected by external environmental factors such as lighting or occlusion, ensuring the stability and reliability of the motion coordinate data. Compared with visual positioning, forward kinematics calculation has a higher data update frequency and lower latency, and can provide real-time feedback on the precise position of the robotic arm's end effector, providing reliable reference data for the subsequent establishment of the mapping matrix.

[0036] Step S13: Determine the mapping matrix from the preset image coordinate system to the robotic arm motion coordinate system based on the image coordinates and the motion coordinates.

[0037] In this step, the linear transformation relationship can be solved by using the least squares method to obtain the mapping matrix, which describes the complete geometric transformation relationship from the image coordinate system to the robot arm motion coordinate system.

[0038] In some embodiments, when performing step S13, a first coordinate point set and a second coordinate point set are first generated respectively. The first coordinate point set includes the image coordinates of multiple target feature points, and the second coordinate point set includes the motion coordinates of multiple target feature points. Based on the first coordinate point set, the second coordinate point set, and the linear transformation relationship from the image coordinate system to the robot arm motion coordinate system, an overdetermined system of equations is constructed, wherein the linear transformation relationship includes rotation matrix components and translation vector components. The overdetermined system of equations is solved using the least squares method to obtain the mapping matrix from the preset image coordinate system to the robot arm motion coordinate system.

[0039] In this step, in the touch testing scenario, the product under test is typically considered a plane. Therefore, the mapping from the image coordinate system to the robotic arm motion coordinate system is a planar mapping from a two-dimensional plane to three-dimensional space, rather than a general three-dimensional spatial transformation. Let the two-dimensional point in the image coordinate system be... That is, the pixel coordinates in the image, and the corresponding three-dimensional points in the robot arm's motion coordinate system. This refers to the three-dimensional position of the robotic arm's end effector in space. Since touch operations occur on the touch surface of the product under test, and the end effector needs to be precisely aligned with a target point on this plane during robotic arm movement, a mapping relationship from the image plane to the robotic arm's motion plane needs to be established. When the touch surface of the product under test is approximately planar, this mapping can be simplified to a two-dimensional to three-dimensional mapping model, as detailed below: ; ; ; in, These are the coordinates of the target feature point or target touch point in the image coordinate system. The coordinates of this point in the coordinate system of the robotic arm's motion are given. Let be the mapping parameters to be solved. The three equations above describe the image coordinates respectively. The linear mapping relationship between the three spatial dimensions X, Y, and Z of the robotic arm's motion coordinate system. Each point on the image plane. This linear transformation maps the data to a three-dimensional point in the robotic arm's motion space. This mapping model essentially maps image plane coordinates... First, map the touch plane to a two-dimensional coordinate system coplanar with the touch plane, i.e., the corresponding plane of the touch plane in the robotic arm space, and then combine this with the spatial position of the plane in the robotic arm space. This yields complete three-dimensional coordinates. For curved screens with small curvature, this linear planar model still applies as a first-order approximation, with errors within an acceptable range. Therefore, this method can also be applied to micro-curved surface products.

[0040] Image coordinates of multiple target feature points and corresponding motion coordinates Substituting these equations into the three equations above, we obtain three independent systems of equations: For the X coordinate: This constitutes a question about the unknown. A system of n equations; for the Y coordinate: This constitutes a question about the unknown. A system of n equations; for the Z coordinate: Constitute a number of unknowns A system of n equations. Where i = 1, 2, ..., n, and n is the number of feature points.

[0041] When n>3, each system of equations becomes overdetermined, with more equations than unknowns, and no exact solution exists. In this case, the least squares method is used to minimize the sum of squared mapping errors at all feature points. Taking the system of equations for the X-coordinate as an example, the least squares solution involves finding a set of... , so that the objective function Take the minimum value, and then... Taking the partial derivatives and setting them to zero yields a 3×3 system of linear equations. Solving this system of linear equations will give us the solution. The optimal solution can be found similarly. and The values ​​of are determined by arranging the nine obtained parameters in rows and columns to obtain the mapping matrix from the image coordinate system to the robot arm motion coordinate system. The mapping matrix M describes the complete linear transformation relationship from the planar coordinates of the image to the three-dimensional coordinates of the robotic arm. Through this mapping matrix, any visible touch point on the image... The three-dimensional coordinates of each component in the robotic arm's motion coordinate system can be obtained through the following calculations: ; ; By collecting the corresponding coordinates of at least four feature points, an overdetermined system of equations containing redundant information is constructed. The least squares method is then used to solve this system, minimizing the overall fitting error of all corresponding point pairs. This effectively suppresses the impact of single-point measurement noise on mapping accuracy. Simultaneously, the redundant information provided by the n feature points ensures that the solution to the mapping matrix is ​​statistically optimal. Even if there are slight deviations in the coordinate measurements of individual feature points, it will not significantly affect the final mapping accuracy, guaranteeing the robustness and accuracy of the mapping matrix and solving the problem of coordinate mapping deviations caused by tooling assembly errors, product placement offsets, etc.

[0042] Step S14: Obtain the first coordinates of the touch point to be tested in the preset image coordinate system, and map the first coordinates to the second coordinates of the touch point to be tested in the robotic arm motion coordinate system according to the mapping matrix.

[0043] In this step, the image coordinates of the touch point to be tested are usually pre-marked on the image according to the test requirements, such as the center of the screen, the position of a specific button, etc. The first coordinate is substituted into the mapping matrix for transformation to obtain the robotic arm motion coordinates of the touch point in the actual physical space, i.e., the second coordinate. By accurately converting the first coordinate in the test plan into the second coordinate that the robotic arm can execute, an accurate correspondence between the theoretical test point and the actual executed action is achieved.

[0044] Step S15: Control the robotic arm to perform a touch operation on the product under test according to the second coordinate.

[0045] In this step, the robotic arm is precisely moved to the coordinate position according to the second coordinate to perform the touch operation. This ensures that every touch operation of the robotic arm can be accurately applied to the preset test point, eliminating the touch position deviation caused by product placement offset or the positioning error of the robotic arm itself, and fundamentally guaranteeing the accuracy and repeatability of the test results.

[0046] In some embodiments, when performing step S15, the basic gestures corresponding to the touch operation are first determined according to the test requirements of the product under test. The basic gestures include at least one of single click, double click, long press, straight swipe, curved swipe, edge touch, multi-finger zoom, multi-finger rotation, and fine writing. Then, the gesture parameters corresponding to each basic gesture are determined, including spatial motion trajectory, motion speed, and touch pressure. A trajectory command stream is generated according to the execution order of the basic gestures and the gesture parameters. The robotic arm is controlled to perform touch operations on the product under test according to the second coordinates and the trajectory command stream.

[0047] In this step, based on the testing requirements of the product under test, suitable basic gestures are selected from a preset gesture library. Examples include single-point touch testing (click for single-point touch), trajectory linearity testing (swipe for trajectory linearity testing), and multi-point synchronization testing (multi-finger zoom for multi-point synchronization testing). Spatial motion trajectory, speed, and touch pressure parameters are set for each gesture, and these are sequentially combined according to a specified execution order to generate a trajectory command stream executable by the robotic arm. Finally, the trajectory command stream is combined with the target coordinates to drive the robotic arm to perform touch operations. Through standardized management of the gesture library and automatic generation of trajectory command streams, diverse testing requirements are transformed into specific actions executable by the robotic arm. This achieves flexibility in test scheme configuration and a high degree of automation in robotic arm execution. It can adapt to different types of touch product testing without manual programming, significantly improving the versatility and ease of use of the testing system.

[0048] In some embodiments, when the robotic arm is controlled to perform touch operations on the product under test according to the second coordinate and trajectory command flow, the robotic arm is first controlled to move to the second physical position corresponding to the second coordinate, so that the touch head of the robotic arm contacts the touch surface at the second physical position; then, the touch pressure value of the touch head of the robotic arm is collected at a preset sampling frequency, and the pressure deviation value between the touch pressure value and the preset target pressure value is determined; the height adjustment amount and posture adjustment amount of the touch head of the robotic arm are determined according to the pressure deviation value, and the height adjustment amount and posture adjustment amount of the touch head of the robotic arm are respectively adjusted.

[0049] In this step, the pressure applied during the contact between the robotic arm's touch head and the touch surface of the product under test directly affects the triggering stability of the touch signal and the consistency of the test results. Insufficient pressure may prevent the touch from being effectively recognized, while excessive pressure may damage the product's touch panel or introduce additional interference. Therefore, it is necessary to monitor and automatically adjust the touch pressure in real time. The system has a preset target pressure value. The target pressure value is preset according to the type of product being tested and the testing requirements. For example, for a smartphone touchscreen, the target pressure value can be set to 1.0N. During the touch operation performed by the robotic arm, the force control sensor collects the actual touch pressure value between the touch head and the touch surface in real time at a preset high sampling frequency. Actual touch pressure value With target pressure value The difference between them is the pressure deviation value. When e > 0, it indicates that the actual touch pressure value is less than the target pressure value, and the touch head needs to press further down towards the touch surface; when e < 0, it indicates that the actual touch pressure value is greater than the target pressure value, and the touch head needs to be lifted appropriately away from the touch surface; when e = 0, it indicates that the actual touch pressure value has reached the target pressure value, and no adjustment is needed. After obtaining the pressure deviation value e, the system needs to convert it into specific adjustment actions at the end of the robotic arm, involving adjustments in two dimensions.

[0050] Height adjustment amount This refers to the displacement adjustment of the robotic arm's end effector along a direction perpendicular to the touch plane, used to address issues of excessive or insufficient pressure. Height adjustment. The conversion relationship between the pressure deviation value e and the pressure deviation value e is as follows: ,in, This is the preset first conversion factor, representing the height adjustment amount corresponding to a unit pressure deviation. When e is positive... When e is positive, the robotic arm's end effector moves downwards, increasing the touch pressure; when e is negative, If the value is negative, the robotic arm's end effector moves upward, reducing the touch pressure. Direct correction of touch pressure is achieved through height adjustment. (Attitude adjustment amount) This refers to the angle adjustment amount of the robotic arm's end effector around the touch contact point, used to address the issue of misalignment in the contact angle between the touch head and the touch surface. In actual testing, if the touch head at the robotic arm's end effector does not contact the touch surface at an angle perpendicular to it—for example, due to a curved surface or deformation of a flexible screen—then even if the height is accurate, the contact area and pressure distribution of the touch head will be uneven, affecting the accuracy of touch recognition. (Attitude adjustment amount) The conversion relationship between the pressure deviation value e and the pressure deviation value e is as follows: ,in The first and second conversion factors are preset and represent the attitude adjustment amount corresponding to a unit pressure deviation. When the pressure deviation reflects an abnormal contact area, the system adjusts the angle to achieve the optimal contact posture between the touch head and the touch surface. The first and second conversion factors can be calibrated and determined experimentally.

[0051] The robotic arm controller receives the height adjustment amount. and attitude adjustment amount Then, the end effector of the robotic arm performs corresponding displacement and angle adjustments. After adjustment, the force control sensor collects new actual pressure values ​​again and calculates new pressure deviations. If the deviation is still outside the allowable range, the above adjustment process is repeated, forming a closed-loop feedback control of acquisition, calculation, adjustment, and re-acquisition. Through the above closed-loop feedback control, even if there are slight deformations on the surface of the tested product, such as changes in the curvature of a curved screen, indentations of a flexible screen after pressure, slight vibrations during robotic arm movement, or slight deviations in tooling assembly, the system can respond in real time and dynamically correct the end effector position and posture, ultimately ensuring that the actual touch pressure stabilizes and converges to the target pressure value. This step achieves real-time monitoring and dynamic adjustment of touch pressure through high-frequency closed-loop feedback, controlling pressure fluctuations within a very small range. This effectively overcomes the contact instability caused by deformation of curved and flexible screens, ensuring pressure consistency throughout the touch process. The test results are unaffected by physical factors such as robotic arm movement vibrations or product surface deformation, significantly improving test accuracy and consistency.

[0052] In some embodiments, after adjusting the height and posture of the robotic arm's touch head according to the height adjustment amount and posture adjustment amount respectively, the touch coordinate sequence and touch time sequence reported by the product under test can be obtained respectively; based on the touch coordinate sequence, the touch accuracy and trajectory linearity of the current touch operation are determined; based on the touch time sequence, the multi-point synchronization error and response delay time of the current touch operation are determined; finally, the current test result is determined based on the touch accuracy, trajectory linearity, multi-point synchronization error, and response delay time.

[0053] In this step, the touch coordinate sequence is the set of touch position coordinates continuously reported by the touch chip inside the product under test during touch operation; the touch time sequence is the set of timestamps corresponding to each touch coordinate report. The touch coordinate sequence and touch time sequence reported by the product under test are collected synchronously. For touch accuracy indicators, the root mean square value of the Euclidean distance between the actual reported coordinates and the theoretical coordinates can be calculated to determine the accuracy. ;in, Let i be the actual coordinates of the i-th reporting point. For the corresponding theoretical coordinates, The distance is Euclidean. For the trajectory linearity index, the coordinates of the reported points on the actual sliding trajectory are fitted to the ideal trajectory curve using the least squares method, and the root mean square value of the vertical distance from each reported point to the ideal trajectory is calculated to determine the linearity. ;in, denoted as , where is the vertical distance from the i-th reporting point to the fitted curve.

[0054] For the multi-point synchronization error index, the reporting time of each finger in multi-finger touch is collected, and the maximum time difference is calculated: ;in, Let be the reporting time of the i-th reference. For the response latency metric, calculate the time difference between the touch operation occurrence and the reporting time: ,in, For reporting the time of the reporting point, This is the moment when a touch operation is initiated. By upgrading the traditional qualitative evaluation method based on subjective human judgment to a data-driven evaluation method based on mathematical quantification models, it is possible to accurately quantify various performance indicators such as touch accuracy, trajectory linearity, multi-point synchronization error, and response latency. Each indicator has a clear physical meaning and calculation formula, and the test results are objective, traceable, and can accurately locate the cause of product defects.

[0055] In this automated testing method for capacitive sensing products, a full-area image of the product under test is acquired, and the image coordinates of target feature points are extracted. This digitizes the actual placement position of the product, providing a data foundation for coordinate mapping. By controlling a robotic arm to touch the feature points and determining their motion coordinates in the robotic arm's own coordinate system, the spatial position corresponding to the image is reproduced in the robotic arm's own coordinate system. The mapping matrix between the two coordinate systems is determined using the correspondence between image coordinates and motion coordinates. The image coordinates of the touch point under test are then substituted into the mapping matrix to convert them into motion coordinates, ensuring an accurate correspondence between the theoretical test point and the actual action. Controlling the robotic arm to perform touch operations based on the motion coordinates eliminates touch position deviations caused by product placement offsets or robotic arm positioning errors, ensuring the accuracy and repeatability of the test results.

[0056] To implement the automated testing method for capacitive sensing products corresponding to the above method embodiments, in order to achieve the corresponding functions and technical effects. See also Figure 2 , Figure 2 This diagram illustrates the structure of an automated testing device for capacitive sensing products according to an embodiment of this application. For ease of explanation, only the parts relevant to this embodiment are shown. The automated testing device for capacitive sensing products provided in this embodiment includes a first determining module 21, a second determining module 22, a third determining module 23, a mapping module 24, and a touch module 25, as detailed below: The first determining module 21 is used to acquire a global image of the product under test and determine the image coordinates of the target feature point in the global image under a preset image coordinate system; the second determining module 22 is used to control the robotic arm to move to the first physical position corresponding to the target feature point and determine the motion coordinates of the target feature point under the robotic arm motion coordinate system; the third determining module 23 is used to determine a mapping matrix from the preset image coordinate system to the robotic arm motion coordinate system based on the image coordinates and the motion coordinates; the mapping module 24 is used to acquire the first coordinates of the touch point to be tested under the preset image coordinate system and map the first coordinates to the second coordinates of the touch point to be tested under the robotic arm motion coordinate system based on the mapping matrix; the touch module 25 is used to control the robotic arm to perform touch operation on the product under test based on the second coordinates.

[0057] The automated testing apparatus for capacitive sensing products described above can implement the automated testing method for capacitive sensing products described in the above method embodiments. The options in the above method embodiments are also applicable to this embodiment, and will not be detailed here. The remaining content of this application's embodiments can be referred to the content of the above method embodiments, and will not be repeated in this embodiment.

[0058] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 3 As shown, the computer device 30 of this embodiment includes: at least one processor 31 ( Figure 3 (Only one is shown in the diagram), memory 32, and computer program 33 stored in the memory 32 and executable on the at least one processor 31, wherein the processor 31 executes the computer program 33 to implement the steps in any of the above method embodiments.

[0059] The computer device 30 may be a desktop computer or a cloud server, etc. The computer device 30 may include, but is not limited to, a processor 31 and a memory 32. Those skilled in the art will understand that... Figure 3 The computer device 30 is merely an example and does not constitute a limitation on the computer device 30. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0060] The processor 31 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0061] In some embodiments, the memory 32 may be an internal storage unit of the computer device 30, such as a hard disk or memory of the computer device 30. In other embodiments, the memory 32 may be an external storage device of the computer device 30, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 30. Furthermore, the memory 32 may include both internal and external storage units of the computer device 30. The memory 32 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program 33. The memory 32 may also be used to temporarily store data that has been output or will be output.

[0062] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above method embodiments.

[0063] This application provides a computer program product that, when run on a computer device, enables the computer device to execute the steps described in the various method embodiments above.

[0064] In the several embodiments provided in this application, it will be understood that each block in the flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0065] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0066] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application for those skilled in the art.

Claims

1. An automated testing method for capacitive sensing products, characterized in that, include: Acquire a full-area image of the product under test, and determine the image coordinates of the target feature points in the full-area image in a preset image coordinate system; Control the robotic arm to move to the first physical position corresponding to the target feature point, and determine the motion coordinates of the target feature point in the robotic arm motion coordinate system; Based on the image coordinates and the motion coordinates, determine the mapping matrix from the preset image coordinate system to the robotic arm motion coordinate system; Obtain the first coordinates of the touch point to be tested in the preset image coordinate system, and map the first coordinates to the second coordinates of the touch point to be tested in the robot arm motion coordinate system according to the mapping matrix; The robotic arm is controlled to perform touch operations on the product under test according to the second coordinate.

2. The automated testing method for capacitive sensing products as described in claim 1, characterized in that, Determining the image coordinates of the target feature points in the global image in the preset image coordinate system includes: Distortion correction and edge contour extraction are performed on the global image to identify the effective touch area; Within the effective touch area, target feature points that meet preset feature recognition conditions are identified. The preset feature recognition conditions include corner response conditions and edge gradient direction change conditions. Based on the grayscale distribution of the target feature points in the local area, subpixel-level position fitting is performed on the target feature points to obtain the image coordinates corresponding to the target feature points in the preset image coordinate system.

3. The automated testing method for capacitive sensing products as described in claim 1, characterized in that, The process of controlling the robotic arm to move to the first physical position corresponding to the target feature point, and determining the motion coordinates of the target feature point in the robotic arm's motion coordinate system, includes: Control the robotic arm to move to the first physical position corresponding to the target feature point, so that the robotic arm touch head contacts the touch surface at the first physical position; Read the encoders of each joint of the robotic arm to obtain multiple joint code values; Based on the forward kinematics model of the robotic arm, kinematic calculations are performed on the multiple joint encoding values ​​to obtain the motion coordinates of the target feature point in the robotic arm's motion coordinate system.

4. The automated testing method for capacitive sensing products as described in claim 1, characterized in that, The step of determining the mapping matrix from the preset image coordinate system to the robotic arm motion coordinate system based on the image coordinates and the motion coordinates includes: A first set of coordinate points and a second set of coordinate points are generated respectively, wherein the first set of coordinate points includes the image coordinates of multiple target feature points, and the second set of coordinate points includes the motion coordinates of multiple target feature points; Based on the first set of coordinate points, the second set of coordinate points, and the linear transformation relationship from the image coordinate system to the robot arm motion coordinate system, an overdetermined system of equations is constructed, wherein the linear transformation relationship includes rotation matrix components and translation vector components. The overdetermined equations are solved using the least squares method to obtain the mapping matrix from the preset image coordinate system to the robotic arm motion coordinate system.

5. The automated testing method for capacitive sensing products as described in claim 1, characterized in that, The step of controlling the robotic arm to perform a touch operation on the product under test according to the second coordinate includes: The basic gestures corresponding to the touch operation are determined according to the test requirements of the product under test. The basic gestures include at least one of the following: single click, double click, long press, straight swipe, curved swipe, edge touch, multi-finger zoom, multi-finger rotation, and fine writing. Determine the gesture parameters corresponding to each of the basic gestures, including spatial motion trajectory, motion speed, and touch pressure; A trajectory instruction stream is generated based on the execution order of the basic gestures and the gesture parameters; The robotic arm is controlled to perform touch operations on the product under test based on the second coordinates and the trajectory command flow.

6. The automated testing method for capacitive sensing products as described in claim 5, characterized in that, The step of controlling the robotic arm to perform touch operations on the tested product based on the second coordinates and the trajectory command flow includes: Control the robotic arm to move to the second physical position corresponding to the second coordinate, so that the robotic arm touch head contacts the touch surface at the second physical position; The touch pressure value of the robotic arm's touch head is collected at a preset sampling frequency, and the pressure deviation value between the touch pressure value and the preset target pressure value is determined. The height adjustment amount and posture adjustment amount of the robotic arm touch head are determined based on the pressure deviation value, and the height adjustment amount and posture adjustment amount of the robotic arm touch head are respectively performed based on the height adjustment amount and the posture adjustment amount.

7. The automated testing method for capacitive sensing products as described in claim 6, characterized in that, After adjusting the height and attitude of the robotic arm's touch head according to the height adjustment amount and the attitude adjustment amount respectively, the method further includes: Obtain the touch coordinate sequence and touch time sequence reported by the product under test, respectively; Based on the touch coordinate sequence, determine the touch accuracy and trajectory linearity of the current touch operation; Based on the touch time sequence, determine the multi-point synchronization error and response delay time of the current touch operation; The current test result is determined based on the touch accuracy, the trajectory linearity, the multi-point synchronization error, and the response delay time.

8. An automated testing device for capacitive sensing products, characterized in that, include: The first determining module is used to acquire a full-domain image of the product under test and determine the image coordinates of the target feature points in the full-domain image in a preset image coordinate system. The second determining module is used to control the robotic arm to move to the first physical position corresponding to the target feature point, and to determine the motion coordinates of the target feature point in the robotic arm motion coordinate system. The third determining module is used to determine a mapping matrix from the preset image coordinate system to the robotic arm motion coordinate system based on the image coordinates and the motion coordinates. The mapping module is used to obtain the first coordinates of the touch point to be tested in the preset image coordinate system, and to map the first coordinates to the second coordinates of the touch point to be tested in the robot arm motion coordinate system according to the mapping matrix; The touch module is used to control the robotic arm to perform touch operations on the product under test according to the second coordinate.

9. A computer device, characterized in that, It includes a processor and a memory, the memory being used to store a computer program, which, when executed by the processor, implements the automated testing method for capacitive sensing products as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements an automated testing method for capacitive sensing products as described in any one of claims 1 to 7.

Images