Software testing method and device of terminal equipment, electronic equipment and storage medium

Abstract

The invention discloses a software testing method and device of terminal equipment, electronic equipment and a storage medium. The method comprises the steps that a target motion path from the current position of a mechanical arm to a target position is determined; wherein the target position is the position of a to-be-measured control in the terminal equipment; and the mechanical arm moves to the target position according to the target motion path, so that the mechanical arm executes a touch control test task on the to-be-measured control piece. Through the mode, the test efficiency and accuracy of the terminal equipment can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of device testing, and particularly to a software testing method, apparatus, electronic device, and storage medium for a terminal device. Background Art

[0002] Currently, with the popularization of terminal devices such as smart large screens, smartphones, and tablets, software testing in terminal devices has become an important link in the product development process.

[0003] Traditional software testing methods mainly rely on manual operations or script-based automated testing. However, these methods have some limitations: manual testing is costly, inefficient, and prone to fatigue and errors; although script-based automated testing can improve efficiency, it requires writing a large number of test scripts and is difficult to adapt to frequent changes in software interfaces. Summary of the Invention

[0004] The main technical problem to be solved by the present application is to provide a software testing method, apparatus, electronic device, and storage medium for a terminal device, which can improve the testing efficiency and accuracy of the terminal device.

[0005] To solve the above technical problem, one technical solution adopted by the present application is: to provide a software testing method for a terminal device, the method includes: determining a target movement path from the current position of the robotic arm to the target position; wherein, the target position is the position of the control to be tested in the terminal device; using the robotic arm to move to the target position according to the target movement path, so that the robotic arm performs a touch test task on the control to be tested.

[0006] To solve the above technical problem, another technical solution adopted by the present application is: to provide a software testing apparatus for a terminal device, the apparatus includes a determination module and a testing module; the determination module is used to determine a target movement path from the current position of the robotic arm to the target position; wherein, the target position is the position of the control to be tested in the terminal device; the testing module is used to use the robotic arm to move to the target position according to the target movement path, so that the robotic arm performs a touch test task on the control to be tested.

[0007] To solve the above technical problem, another technical solution adopted by the present application is: to provide an electronic device, the electronic device includes a processor and a memory, the memory stores program instructions, and the processor is used to execute the program instructions to implement the above-mentioned testing method for the terminal device.

[0008] To solve the above technical problem, another technical solution adopted by the present application is: to provide a computer-readable storage medium, the computer-readable storage medium is used to store program instructions, and the program instructions can be executed to implement the above-mentioned testing method for the terminal device.

[0009] In the above technical solution, since the robotic arm has advantages such as high precision, high efficiency, and good repeatability, using the robotic arm to perform touch tests on the control to be tested in the terminal device can improve the test efficiency and accuracy of the terminal device. In addition, the method of using the robotic arm to test the terminal device is a non-intrusive fast exploration test, which can perform touch tests on the terminal device without interfering with the normal operation of the terminal device, improving the safety and reliability of the test. Moreover, using the robotic arm to perform touch tests on the control to be tested in the terminal device can reduce the participation of manual labor and lower the test cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a schematic flowchart of an embodiment of the software testing method for a terminal device provided by the present application;

[0011] Figure 2 is Figure 1 a schematic flowchart of an embodiment of step S11 shown;

[0012] Figure 3 is a schematic flowchart of another embodiment of the software testing method for a terminal device provided by the present application;

[0013] Figure 4 is Figure 3 a schematic flowchart of an embodiment of step S34 shown;

[0014] Figure 5 is Figure 4 a schematic flowchart of an embodiment of step S41 shown;

[0015] Figure 6 is Figure 5 a schematic flowchart of an embodiment of step S52 shown;

[0016] Figure 7 is Figure 6 a schematic flowchart of an embodiment of step S61 shown;

[0017] Figure 8 is Figure 4 a schematic flowchart of an embodiment of step S43 shown;

[0018] Figure 9 is Figure 8 a schematic flowchart of an embodiment of step S82 shown;

[0019] Figure 10 is Figure 4 a schematic flowchart of an embodiment of step S44 shown;

[0020] Figure 11 is a schematic structural diagram of an embodiment of the software testing device for a terminal device provided by the present application;

[0021] Figure 12 It is a schematic structural diagram of an embodiment of an electronic device provided by the present application;

[0022] Figure 13 It is a schematic structural diagram of an embodiment of a computer-readable storage medium provided by the present application. Specific embodiments

[0023] The following will combine the accompanying drawings of the specification to elaborate on the solutions of the embodiments of the present application in detail.

[0024] In the following description, specific details such as specific system structures, interfaces, and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the present application.

[0025] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after. In addition, "plurality" in this article means two or more. In addition, the term "at least one" in this article represents any one of a plurality or any combination of at least two of a plurality. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set composed of A, B, and C.

[0026] Please refer to Figure 1 , Figure 1 It is a schematic flowchart of an embodiment of a software testing method for a terminal device provided by the present application. It should be noted that if there are substantially the same results, this embodiment is not limited to Figure 1 the process sequence shown. As Figure 1 shown, this embodiment includes:

[0027] Step S11: Determine the target motion path from the current position of the robotic arm to the target position.

[0028] In this embodiment, the target motion path from the current position of the robotic arm to the target position is determined; wherein, the target position is the position of the control to be tested in the terminal device. That is to say, planning a motion path for the robotic arm from the current position to the position of the control to be tested in the terminal device can ensure that the robotic arm will not collide with the terminal device or other objects when performing the touch test task on the control to be tested, thus ensuring that the robotic arm can stably and reliably perform the touch test task on the control to be tested subsequently.

[0029] In one embodiment, after determining the initial motion path from the current position of the robotic arm to the target position, the initial motion path can be directly used as the target motion path.

[0030] Of course, in other embodiments, after determining the initial motion path from the current position of the robotic arm to the target position, the initial motion path can be smoothed, and the smoothed initial motion path can be used as the target motion path. On the one hand, the unsmoothed initial motion path may have sharp turns or mutation points, resulting in unnecessary jitter and oscillation during the movement of the robotic arm; by smoothing the initial motion path, these phenomena can be significantly reduced, making the movement of the robotic arm more stable. On the other hand, the smoothed initial motion path can ensure that the robotic arm does not generate excessive mechanical shock due to sudden direction changes or speed changes during the movement, thereby protecting the components of the robotic arm from damage. On the other hand, the smoothed initial motion path can optimize the motion trajectory of the robotic arm, reduce unnecessary motion paths, and thus improve the motion efficiency.

[0031] Step S12: Use the robotic arm to move to the target position according to the target motion path, so that the robotic arm performs a touch test task on the control to be tested.

[0032] In this embodiment, the robotic arm is used to move to the target position according to the target motion path, so that the robotic arm performs a touch test task on the control to be tested. Since the target position is the position of the control to be tested in the terminal device, after the robotic arm moves to the target position according to the target motion path, the control to be tested can be tested.

[0033] Since the robotic arm has advantages such as high precision, high efficiency, and good repeatability, using the robotic arm to perform a touch test on the control to be tested in the terminal device can improve the test efficiency and accuracy of the terminal device. In addition, the test method using the robotic arm for the terminal device is a non-intrusive fast exploration test, which can perform a touch test on the terminal device without disturbing the normal operation of the terminal device, improving the safety and reliability of the test. Furthermore, using the robotic arm to perform a touch test on the control to be tested in the terminal device can reduce the participation of humans and lower the test cost.

[0034] Please refer to Figure 2 , Figure 2 which Figure 1 is a schematic flowchart of an embodiment of step S11 shown. It should be noted that if there are substantially the same results, this embodiment is not limited to Figure 2 the Figure 2 flow sequence shown. As

[0035] Step S21: Obtain the initial motion path from the current position of the robotic arm to the target position.

[0036] In this embodiment, the initial motion path from the current position of the robotic arm to the target position is obtained.

[0037] In one embodiment, determining the initial motion path from the current position of the robotic arm to the target position specifically includes the following steps:

[0038] Step 1: Initialize the RRT tree. Specifically, select a starting point X start , the current position of the robotic arm; create a tree T that contains X start .

[0039] Step 2: Random sampling. Specifically, randomly select a point X rand in the configuration space, where the configuration space is the set of all possible robotic arm joint angles, and each point represents a set of joint angles.

[0040] Step 3: Find the nearest neighbor node. Specifically, find the node X rand in the tree T that is closest to X near , where the distance can be calculated using the Euclidean distance as follows:

[0041]

[0042] where: n represents the number of joints.

[0043] Step 4: Expand the tree. Specifically, expand in the direction from X near to X rand to generate a new node X new , where during the expansion process, it is necessary to ensure that the new node is in a collision-free workspace, that is, it will not collide with the terminal device or other objects, and limit the maximum distance δ between X new and X near to control the growth rate of the tree T.

[0044] Step 5: Add the new node to the tree T. Specifically, if X new is a valid collision-free state, then add it to the tree T.

[0045] Step 6: Check if the target is reached. Specifically, if X new is close enough to the target position X goal , then stop the search. Otherwise, repeat steps two to five until the preset number of iterations or time limit is reached.

[0046] Step 7: Extract the path. Specifically, once the target position is found, by tracing the parent nodes in the tree T, from X goalBacktrack to X start , construct a collision-free path from the current position to the target position as the initial motion path of the robotic arm from the current position to the target position.

[0047] In a specific embodiment, before determining the initial motion path of the robotic arm from the current position to the target position, the working range required by the robotic arm and the safe position of the robotic arm are calculated. Among them, the working range required by the robotic arm can be understood as the spatial area that the end effector of the robotic arm can reach.

[0048] The screen size of the terminal device is width W and height H. In order to enable the robotic arm to reach every corner of the terminal device, it is necessary to ensure that the working range of the robotic arm can at least cover the distances from the center of the screen to the four corner points. Since the distances from the two endpoints of the screen diagonal to the center of the screen are the largest, the working range of the robotic arm is determined by calculating the length of the screen diagonal.

[0049] Specifically, the screen diagonal length Therefore, the working range of the robotic arm needs to be at least equal to D.

[0050] The base of the robotic arm needs to be installed at a position no more than D / 2 away from the center of the terminal device to ensure that the working range of the robotic arm can cover the entire screen of the terminal device. In addition, it is also necessary to ensure that the robotic arm will not collide with the terminal device or other objects during the movement process, so as to ensure that the robotic arm can subsequently perform the touch test task on the control to be measured stably and reliably.

[0051] Step S22: Smooth the initial motion path to obtain the target motion path.

[0052] In this embodiment, the initial motion path is smoothed to obtain the target motion path. Since the target motion path is obtained by smoothing the initial motion path, there will be no sharp turns or mutation points in the target motion path. Subsequently, the robotic arm will not generate unnecessary jitters and oscillations during the movement according to the target motion path, making the movement of the robotic arm smoother. In addition, since the target motion path is obtained by smoothing the initial motion path, the robotic arm will not generate excessive mechanical shocks due to sudden direction changes or speed changes during the subsequent movement according to the target motion path, thereby protecting the components of the robotic arm from damage. Furthermore, since the target motion path is obtained by smoothing the initial motion path, that is, the target motion path is an optimized motion path, the robotic arm will be able to reduce unnecessary motion paths during the subsequent movement according to the target motion path, thereby improving the motion efficiency and further improving the touch test efficiency of the robotic arm for the control to be measured in the terminal device.

[0053] In one embodiment, path smoothing is performed on the initial motion path to obtain the target motion path, which specifically includes the following steps:

[0054] Step 1: Select the order of the Bezier curve. Specifically, an nth-order Bezier curve is defined by n + 1 control points.

[0055] Step 2: Define the control points. Specifically, select the key points on the initial motion path as the control points.

[0056] Step 3: Calculate the Bezier curve. Specifically, the formula for the point P(t) on the Bezier curve is as follows:

[0057]

[0058] where P i represents the control point; B i,n (t) represents the Bernstein polynomial of order n, defined as: t is a parameter between 0 and 1, representing the relative position along the Bezier curve.

[0059] Step 4: Fit the Bezier curve. Specifically, first, the objective function E is the sum of the squares of the distances between the points on the initial motion path and the corresponding points on the Bezier curve. There are m points on the initial motion path as {Q1, Q2,..., Q m}, and the corresponding points on the Bezier curve are {P(t1), P(t2),..., P(t m )}, then the objective function can be defined as:

[0060]

[0061] where ||·|| represents the Euclidean norm of the vector.

[0062] Then, minimize the objective function E to find the optimal control points {P0, P1,..., P n}.

[0063] Then, use the gradient descent method to gradually optimize the positions of the control points. Among them, in each iteration step, according to the positions of the current control points, calculate the gradient (i.e., partial derivative) of the objective function E with respect to each control point; according to the direction and magnitude of the gradient, adjust the positions of the control points to reduce the value of the objective function E. The iteration process continues until the value of the objective function E converges or reaches the preset number of iterations.

[0064] Among them, in each iteration step, the update of the control points can be performed through the following formula:

[0065]

[0066] Among them, α represents the learning rate, which determines the magnitude of control update in each iteration step; represents the gradient of the objective function E at the control point position.

[0067] Step Five: Verify the target motion path obtained by path smoothing. Specifically, verify the target motion path obtained by smoothing to determine whether the target motion path meets the requirements of the robotic arm movement (such as, whether it meets the kinematic constraints, whether there is a possibility of collision with the terminal device or other objects, etc.); if the target motion path obtained by smoothing meets the requirements of the robotic arm movement, then use the target motion path obtained by smoothing as the final motion path, and the subsequent robotic arm moves according to the target motion path to perform the touch test task on the control to be tested in the terminal device; if the target motion path obtained by smoothing does not meet the requirements of the robotic arm movement, then reselect the order of the Bezier curve or adjust the control points until the obtained target motion path meets the requirements of the robotic arm movement.

[0068] Please refer to Figure 3 , Figure 3 which is a schematic flowchart of another embodiment of the software testing method for the terminal device provided by this application. It should be noted that if there are substantially the same results, this embodiment is not limited to Figure 3 the process sequence shown. As Figure 3 shown, this embodiment includes:

[0069] Step S31: Determine the target motion path from the current position of the robotic arm to the target position.

[0070] Step S31 is similar to Step S11 and will not be elaborated here.

[0071] Step S32: Use the robotic arm to move to the target position according to the target motion path so that the robotic arm performs the touch test task on the control to be tested.

[0072] Step S32 is similar to Step S12 and will not be elaborated here.

[0073] Step S33: Obtain touch test data.

[0074] In this embodiment, touch test data is obtained.

[0075] In one embodiment, the touch test data includes at least one of the following: pixel data of the touch position of the robotic arm, test image data of the touch test process, task information of the touch test task, where the touch test process includes the process of the robotic arm moving to the target position according to the target motion path.

[0076] In a specific embodiment, the touch test data includes image data of the touch test process. Specifically, when the robotic arm moves to the target position according to the target movement path so that the robotic arm performs a touch test task on the control to be tested, the image acquisition device is used to acquire image data, and the image data acquired by the image acquisition device covers the entire process of the robotic arm performing the touch test task.

[0077] In a specific embodiment, the touch test data includes pixel data of the touch position of the robotic arm. Specifically, when the robotic arm moves to the target position according to the target movement path so that the robotic arm performs a touch test task on the control to be tested, the response of the terminal device is monitored in real time, and the pixel data of the touch position of the robotic arm is obtained through the feedback system of the terminal device. The feedback system of the terminal device can detect the contact point between the robotic arm and the terminal device (i.e., the touch position). Among them, the touch position can be represented in the form of coordinates (x, y), where x and y represent the pixel positions in the horizontal and vertical directions respectively.

[0078] In a specific embodiment, the touch test data includes task information of the touch test task. The task information of the touch test task can elaborate on the touch test task currently being performed by the robotic arm. For example, if the robotic arm is clicking the confirmation button on the terminal device, the corresponding task information can be "The robotic arm is clicking the confirmation button on the terminal device".

[0079] In a specific embodiment, the touch test data includes pixel data of the touch position of the robotic arm, test image data of the touch test process, and task information of the touch test task. The timestamps corresponding to the pixel data of the touch position of the robotic arm, the test image data of the touch test process, and the task information of the touch test task are synchronized.

[0080] In a specific embodiment, after the touch test data is obtained, the touch test data is stored, for example, stored in a database, so as to facilitate subsequent analysis of the touch test data to obtain the touch test result.

[0081] Step S34: Analyze the touch test data to obtain the touch test result.

[0082] In this embodiment, the touch test data is analyzed to obtain the touch test result.

[0083] In an embodiment, the touch test result includes at least one of the first test result and the second test result. The first test result is used to characterize whether the test response result of the terminal device is abnormal. The test response result is the response situation of the terminal device based on the touch of the robotic arm on the control to be tested. The second test result is used to characterize whether the robotic arm correctly performs the touch test task.

[0084] When the touch test data includes the pixel data of the touch position of the robotic arm, the test image data of the touch test process, and the task information of the touch test task, and the touch test result includes the first test result, it means that the pixel data of the touch position of the robotic arm, the test image data of the touch test process, and the task information of the touch test task are analyzed to determine whether the test response result of the terminal device is abnormal. That is, by combining multi-modal touch test data, it is verified whether the test response result of the terminal device is abnormal, so as to more accurately verify whether the test response result of the terminal device is abnormal. This helps to discover more potential problems of the terminal device and improve the software quality and reliability of the terminal device.

[0085] When the touch test data includes the pixel data of the touch position of the robotic arm, the test image data of the touch test process, and the task information of the touch test task, and the touch test result includes the second test result, it means that the pixel data of the touch position of the robotic arm, the test image data of the touch test process, and the task information of the touch test task are analyzed to determine whether the robotic arm correctly executes the touch test task. That is, by combining multi-modal touch test data, it is verified whether the robotic arm correctly executes the touch test task, so as to more accurately verify whether the robotic arm correctly executes the touch test task.

[0086] Please refer to Figure 4 , Figure 4 Yes Figure 3 is a schematic flowchart of an embodiment of step S34 shown. It should be noted that if there are substantially the same results, this embodiment is not limited to Figure 4 the flow sequence shown. As Figure 4 shown, the robotic arm includes an end effector for touching the control to be tested, and the touch test data includes the pixel data of the touch position of the robotic arm, the test image data of the touch test process, and the task information of the touch test task. This embodiment includes:

[0087] Step S41: From the test image data of the touch test process, determine the end effector and its corresponding first position, and the control to be tested and its corresponding second position as the first test data.

[0088] In this embodiment, from the test image data of the touch test process, the end effector and its corresponding first position, and the control to be tested and its corresponding second position are determined as the first test data. The robotic arm includes several joints, so that the robotic arm can achieve multi-dimensional degree-of-freedom motion control, adapt to complex working environments and task requirements, thereby improving the test efficiency, test flexibility, and test accuracy of the control to be tested of the terminal device. However, the actual part that contacts the control to be tested of the terminal device is the end effector of the robotic arm. Therefore, the first position corresponding to the end effector of the robotic arm needs to be determined.

[0089] In one embodiment, before determining the end effector and its corresponding first position and the control to be tested and its corresponding second position from the test image data of the touch test process as the first test data, the test image data of the touch test process is preprocessed to improve the quality of the test image data of the touch test process, so that subsequently, based on the test image data of the touch test process, the end effector and its corresponding first position and the control to be tested and its corresponding second position can be accurately determined.

[0090] Among them, the preprocessing of the test image data of the touch test process is not limited and can be specifically set according to actual usage needs. For example, the preprocessing of the test image data of the touch test process can be noise reduction, contrast enhancement, brightness enhancement, grayscale conversion, etc. of the test image data of the touch test process.

[0091] In a specific embodiment, a Gaussian filter is used to denoise the test image data of the touch test process. Among them, the specific formula of the Gaussian function is as follows:

[0092]

[0093] Among them, G(x,y) represents the Gaussian function; σ represents the standard deviation of the Gaussian function, which is used to control the degree of smoothing.

[0094] In a specific embodiment, the contrast of the test image data of the touch test process is enhanced. Among them, the specific formula for contrast enhancement is as follows:

[0095]

[0096] Among them, s k represents the enhanced pixel value; T represents the number of gray levels of the test image data of the touch test process; n represents the total number of pixels of the test image data of the touch test process; n j represents the number of pixels with gray level j.

[0097] In a specific embodiment, the brightness of the test image data of the touch test process is enhanced. Among them, the specific formula for brightness enhancement is as follows:

[0098] s k = k·α + β

[0099] Among them, s k represents the enhanced pixel value; k represents the original pixel value; α represents the brightness enhancement factor; β represents the brightness offset.

[0100] In a specific embodiment, the test image data of the touch test process is grayscaled. Among them, the specific formula for grayscaling is as follows:

[0101] Y = 0.299R + 0.587G + 0.114B

[0102] Wherein, Y represents the grayscale value; R, G, and B are the component values of red, green, and blue in the color image, respectively.

[0103] In one embodiment, as Figure 5 shown, Figure 5 is Figure 4 a schematic flowchart of an embodiment of step S41. From the test image data in the touch test process, the end effector and its corresponding first position, and the control to be tested and its corresponding second position are determined. The specific steps are as follows:

[0104] Step S51: Perform object recognition on the test image data in the touch test process to obtain a number of first reference objects and their corresponding first reference positions.

[0105] In this embodiment, object recognition is performed on the test image data in the touch test process to obtain a number of first reference objects and their corresponding first reference positions. Since the test image data in the touch test process may include objects other than the end effector of the robotic arm and the control to be tested, object recognition is performed on the test image data in the touch test process, and a number of first reference objects and their corresponding first reference positions will be obtained.

[0106] In one embodiment, edge detection and double-threshold detection are sequentially performed on the test image data in the touch test process to obtain a binary image containing edge contours. Among them, in the binary image, the object corresponding to the edge contour is the first reference object, and the position corresponding to the edge contour is the first reference position corresponding to the first reference object. Therefore, edge detection and double-threshold detection are sequentially performed on the test image data in the touch test process to achieve object recognition of the test image data in the touch test process, and a number of first reference objects and their corresponding first reference positions are obtained.

[0107] In a specific embodiment, gradient calculation is performed on the test image data in the touch test process to determine the position of the edge in the test image data in the touch test process. Among them, gradient calculation is performed by calculating the gradient magnitude and direction of each pixel in the test image data in the touch test process. The specific calculation formulas for the gradient magnitude and direction are as follows:

[0108]

[0109] Wherein, G x and G yrespectively represent the gradients of the test image data in the x and y directions during the touch test process; f represents the gray value of the test image data in the touch test process. The result of gradient calculation is an image containing the gradient magnitude and direction.

[0110] Furthermore, the double-threshold detection specifically includes the following steps:

[0111] Step 1: Set two thresholds, a high threshold T h and a low threshold T l .

[0112] Step 2: For each pixel in the test image data during the touch test process, determine whether the pixel is a strong edge, a weak edge, or a non-edge according to the comparison result of its gradient magnitude with the two thresholds. If the gradient magnitude of the pixel is greater than the high threshold T h , then mark the pixel as a strong edge; if the gradient magnitude of the pixel is less than the low threshold T l , then mark the pixel as a non-edge; if the gradient magnitude of the pixel is between the low threshold T l and the high threshold T h , then mark the pixel as a weak edge. For each weak edge pixel in the test image data during the touch test process, check whether there is a strong edge pixel in its 8-neighborhood; if there is a strong edge pixel, then mark the weak edge pixel as a strong edge, otherwise keep it as a weak edge.

[0113] Step 3: Edge connection to obtain a binary image containing the edges in the test image data during the touch test process, where the strong edge pixels are marked as edges, and the weak edge pixels are marked as edges or non-edges according to the strong edge pixels in their neighborhoods. The purpose of edge connection is to remove the pixels that are not edges and retain the pixels on the edges.

[0114] Step S52: Select the end effector and the control to be measured from several first reference objects, and use the first reference position of the first reference object serving as the end effector as the first position corresponding to the end effector, and use the first reference position of the first reference object serving as the control to be measured as the second position of the control to be measured.

[0115] In this embodiment, from several first reference objects, the end effector and the control to be measured are selected, and the first reference position of the first reference object serving as the end effector is used as the first position corresponding to the end effector, and the first reference position of the first reference object serving as the control to be measured is used as the second position of the control to be measured. Since the test image data in the touch test process may include objects other than the end effector of the robotic arm and the control to be measured, object recognition of the test image data in the touch test process will obtain several first reference objects and their corresponding first reference positions. Therefore, it is necessary to further determine the first reference object specifically for the end effector of the robotic arm and the first reference object specifically for the control to be measured from the several first reference objects obtained by recognition. In addition, the first reference position of the first reference object for the end effector of the robotic arm is the position of the end effector of the robotic arm, and the first reference position of the first reference object for the control to be measured is the position of the control to be measured.

[0116] In one embodiment, the coordinates of the minimum bounding rectangle of the first reference object serving as the end effector are used as the first position corresponding to the end effector; and the coordinates of the minimum bounding rectangle of the first reference object serving as the control to be measured are used as the second position corresponding to the control to be measured. The specific representation of the coordinates of the minimum bounding rectangle of the first reference object is as follows:

[0117] Minimum bounding rectangle of the first reference object = (x min , y min , x max , y max )

[0118] Among them, x min , y min , x max , y max respectively represent the minimum x coordinate, minimum y coordinate, maximum x coordinate, and maximum y coordinate of the minimum bounding rectangle of the first reference object.

[0119] In one embodiment, as Figure 6 shown, Figure 6 is Figure 5 a schematic flowchart of an embodiment of step S52. Selecting the end effector and the control to be measured from several first reference objects specifically includes the following sub-steps:

[0120] Step S61: Select at least two first key objects from several first reference objects.

[0121] In this embodiment, at least two first key objects are selected from a number of first reference objects. That is, among the number of first reference objects identified from the test image data in the touch test process, the first reference objects that may be the end effector of the robotic arm and the control to be tested are found and used as the first key objects.

[0122] In one embodiment, as Figure 7 shown, Figure 7 is Figure 6 a schematic flowchart of an embodiment of step S61 shown. Selecting at least two first key objects from a number of first reference objects specifically includes the following sub-steps:

[0123] Step S71: Obtain the key feature points of each first reference object.

[0124] In this embodiment, the key feature points of each first reference object are obtained.

[0125] Step S72: Obtain the preset feature points of at least two preset objects in the database.

[0126] In this embodiment, the preset feature points of at least two preset objects in the database are obtained.

[0127] It should be noted that the execution order of step S71 and step S72 is not limited.

[0128] Step S73: For each first reference object, find the preset feature points that match the key feature points of the first reference object to obtain the matching point pairs corresponding to the first reference object.

[0129] In this embodiment, for each first reference object, find the preset feature points that match the key feature points of the first reference object to obtain the matching point pairs corresponding to the first reference object. In one case, for each first reference object or some first reference objects, matching preset feature points can be found in the database. In another case, only two first key objects can find matching preset feature points in the database.

[0130] In one embodiment, finding the preset feature points that match the key feature points of the first reference object to obtain the matching point pairs corresponding to the first reference object is specifically: obtaining the reference distance between the first feature descriptor of the key feature points of the first reference object and the second feature descriptors of each preset feature point; obtaining at least one preset feature point whose reference distance meets the first distance requirement as the candidate matching point; and selecting a candidate matching point and the key feature points of the first reference object to form the matching point pairs corresponding to the first reference object.

[0131] That is, the preset feature points that form matching point pairs with the key feature points of the first reference object are determined by using the distances between the first feature descriptors of the key feature points of the first reference object and the second feature descriptors of the preset feature points.

[0132] In a specific embodiment, the reference distances between the first feature descriptors of the key feature points of the first reference object and the second feature descriptors of the preset feature points are calculated by using the SURF algorithm.

[0133] In a specific embodiment, the steps for obtaining the first feature descriptors of the key feature points of the first reference object are specifically as follows: obtaining a plurality of object feature points of the first reference object; selecting key feature points from the plurality of object feature points; using the object feature points within a preset distance range of the key feature points to generate the first feature descriptors of the key feature points. That is to say, the first feature descriptors of the key feature points are formed by combining the object feature points within a preset distance range of the key feature points according to certain rules to form a description of the key feature points. In addition, by quickly calculating local features within a certain area around the key feature points, the calculation of the first feature descriptors of the key feature points is accelerated.

[0134] In addition, the integral image can also be used to accelerate the calculation of the first feature descriptors of the key feature points, where the specific formula of the integral image is as follows:

[0135]

[0136] Among them, S(x, y) represents the value of the integral image at the position (x, y); I(i, j) represents the gray value of the image at the position (x, y).

[0137] In a specific embodiment, the object feature points within a preset distance range of the key feature points include: the object feature points other than the key feature points among the plurality of object feature points.

[0138] In a specific embodiment, the step of selecting key feature points from the plurality of object feature points is specifically as follows: obtaining the response values of the respective object feature points; using the object feature points whose response values meet the response requirements as the key feature points. Since the key feature points of the first reference object are the significant feature points of the first reference object, the significant object feature points can be selected as the key feature points by comparing the response values of the respective object feature points of the first reference object and selecting the object feature point with the largest response value as the key feature point.

[0139] In a specific embodiment, a Hessian matrix can be used to determine a plurality of object feature points of the first reference object. The Hessian matrix is a second-order partial derivative matrix that describes the curvature information of the image in a local area. Among them, the calculation formula of the Hessian matrix is specifically as follows:

[0140]

[0141] Among them, I represents the gray value of the image; x and y respectively represent the coordinate axes of the image.

[0142] Step S74: Select at least two first key objects from several first reference objects, where the corresponding matching point pairs meet the matching requirements.

[0143] In this embodiment, at least two first key objects are selected from several first reference objects, where the corresponding matching point pairs meet the matching requirements. In one case, for each first reference object or some first reference objects, a matching preset feature point can be found in the database. At this time, the first reference objects corresponding to the two matching point pairs with the highest similarity are used as the first key objects. In another case, for only two first key objects, a matching preset feature point can be found in the database. At this time, the first reference object for which a matching preset feature point can be found is used as the first key object.

[0144] Step S62: Classify each first key object to determine the end effector and the control to be measured.

[0145] In this embodiment, each first key object is classified to determine the end effector and the control to be measured. Each first key object selected from several first reference objects is classified to determine the end effector of the robotic arm and the control to be measured from the test image data in the touch test process.

[0146] Among them, no limitation is imposed on the category recognition algorithm for classifying each first key object, and it can be specifically set according to actual usage needs.

[0147] In one embodiment, each first key object is classified based on the feature descriptors corresponding to each first key object to determine the end effector of the robotic arm and the control to be measured.

[0148] Step S42: Use the task information of the touch test task to determine the touch test type of the robotic arm as the second test data.

[0149] In this embodiment, the task information of the touch test task is used to determine the touch test type of the robotic arm as the second test data. Since the task information of the touch test task represents the relevant information of the touch test task currently executed by the robotic arm, the touch test type of the robotic arm can be determined based on the task information of the touch test task.

[0150] In one embodiment, the touch test type of the robotic arm is determined by using the task information of the touch test task, specifically: extracting the task key information from the task information of the touch test task; determining the touch test type of the robotic arm based on the task key information. The task key information extracted from the task information of the touch test task is directly related to the touch operation performed by the robotic arm; therefore, determining the touch test type of the robotic arm based on the task key information, that is, determining the touch test type of the robotic arm based on the information directly related to the touch operation performed by the robotic arm, can more accurately determine the touch test type of the robotic arm.

[0151] Of course, in other embodiments, the touch test type of the robotic arm can also be directly determined based on the task information of the touch test task, which is not limited herein.

[0152] In one embodiment, before using the task information of the touch test task to determine the touch test type of the robotic arm as the second test data, the task information of the touch test task is preprocessed to improve the quality of the task information data of the touch test task, so that the touch test type of the robotic arm can be accurately determined based on the task information of the touch test task.

[0153] Among them, the preprocessing is not limited and can be specifically set according to actual usage needs. For example, the preprocessing can be pre-segmentation, stop word removal, etc.

[0154] Step S43: Use the pixel data of the touch position of the robotic arm to determine the touch control actually touched by the end effector as the third test data.

[0155] In this embodiment, the pixel data of the touch position of the robotic arm is used to determine the touch control actually touched by the end effector as the third test data.

[0156] In one embodiment, as Figure 8 shown, Figure 8 is Figure 4 a schematic flowchart of an embodiment of step S43 shown, using the pixel data of the touch position of the robotic arm to determine the touch control actually touched by the end effector, specifically including the following sub-steps:

[0157] Step S81: Perform object recognition on the test image data of the touch test process to obtain a number of second reference objects.

[0158] Step S81 is similar to step S51 and will not be elaborated herein.

[0159] Step S82: Use the pixel data of the touch position of the robotic arm to select a second reference object that matches the touch position from a number of second reference objects as the touch control of the end effector.

[0160] Step S82 is similar to step S52 and will not be elaborated here.

[0161] In one embodiment, as Figure 9 shown, Figure 9 is Figure 8 a schematic flowchart of an embodiment of step S82 shown. The pixel data of the touch position of the robotic arm includes the pixel points of the touch position of the robotic arm. Using the pixel data of the touch position of the robotic arm, a second reference object that matches the touch position is selected from several second reference objects as the touch control of the end effector, which specifically includes the following sub-steps:

[0162] Step S91: Obtain the key feature points of each second reference object.

[0163] Step S91 is similar to step S71 and will not be elaborated here.

[0164] Step S92: Select the feature points to be classified that match the pixel points of the touch position from the key feature points of each second reference object.

[0165] Step S92 is similar to step S73 and will not be elaborated here.

[0166] Step S93: Classify the second reference object corresponding to the feature points to be classified to determine the touch control.

[0167] Step S93 is similar to step S62 and will not be elaborated here.

[0168] Step S44: Analyze the first test data, the second test data, and the third test data to obtain the touch test result.

[0169] In this embodiment, the first test data, the second test data, and the third test data are analyzed to obtain the touch test result. The first test data is determined based on the test image data of the touch test process, the second test data is determined based on the task information of the touch test task, and the third test data is determined based on the pixel data of the touch position of the robotic arm. Therefore, the first test data, the second test data, and the third test data are test data of different modalities. Therefore, by analyzing the first test data, the second test data, and the third test data to obtain the touch test result, that is, by analyzing the multi-modal test data, the obtained touch test result is more accurate.

[0170] In one embodiment, the touch test result includes a first test result. By analyzing the first test data, the second test data, and the third test data, it is determined whether the test response result of the terminal device is abnormal, that is, by analyzing the multimodal test data, it is verified whether the test response result of the terminal device is abnormal. Since the multimodal test data contains sufficient information to describe the test response result of the terminal device, it is possible to accurately verify whether the test response result of the terminal device is abnormal.

[0171] In one embodiment, the touch test result includes a second test result. By analyzing the first test data, the second test data, and the third test data, it is determined whether the robotic arm correctly executes the touch test task, that is, by analyzing the multimodal test data, it is verified whether the robotic arm correctly executes the touch test task. Since the multimodal test data contains sufficient information to describe the execution of the touch test task by the robotic arm, it is possible to accurately verify whether the robotic arm correctly executes the touch test task.

[0172] Please refer to Figure 10 , Figure 10 Yes Figure 4 is a schematic flowchart of an embodiment of step S44 shown. It should be noted that if there are substantially the same results, this embodiment is not limited to Figure 10 the process sequence shown. As Figure 10 shown, this embodiment includes:

[0173] Step S101: Feature extraction is respectively performed on the first test data, the second test data, and the third test data to obtain a first test feature, a second test feature, and a third test feature.

[0174] In this embodiment, feature extraction is respectively performed on the first test data, the second test data, and the third test data to obtain a first test feature, a second test feature, and a third test feature. That is to say, feature extraction will be respectively performed on the first test data, the second test data, and the third test data to obtain the corresponding first test feature, second test feature, and third test feature.

[0175] Step S102: The first test feature, the second test feature, and the third test feature are fused to obtain a fused feature.

[0176] In this embodiment, the first test feature, the second test feature, and the third test feature are fused to obtain a fused feature. That is to say, the first test feature, the second test feature, and the third test feature are combined to obtain a combined feature.

[0177] Step S103: Analyze the fused feature to obtain the touch test result.

[0178] In this embodiment, the fusion feature is analyzed to obtain the touch test result. The first test data is determined based on the test image data of the touch test process, and the first test feature is obtained by feature extraction based on the first test data; the second test data is determined based on the task information of the touch test task, and the second test feature is obtained by feature extraction based on the second test data; the third test data is determined based on the pixel data of the touch position of the robotic arm, and the third test feature is obtained by feature extraction based on the third test data; therefore, the fusion feature obtained by fusing the first test feature, the second test feature, and the third test feature is a multi-modal feature that fuses the features of multi-modal data. Therefore, analyzing the fusion feature to obtain the touch test result, that is, analyzing the multi-modal fusion feature to obtain the touch test result, the obtained touch test result is more accurate.

[0179] In one embodiment, the touch test result includes the second test result; at this time, analyzing the fusion feature to obtain the touch test result is specifically: classifying based on the fusion feature to obtain a classification result; in response to the classification result being the first classification value, it is determined that the robotic arm correctly executes the touch test task; in response to the classification result being the second classification value, it is determined that the robotic arm does not correctly execute the touch test task. That is to say, classifying the fusion feature, when the classification result is the first classification value, it indicates that the robotic arm correctly executes the touch test task, and when the classification result is the second classification value, it indicates that the robotic arm does not correctly execute the touch test task.

[0180] Among them, the magnitudes of the first classification value and the second classification value are not limited and can be specifically set according to actual usage needs. For example, the first classification value is 1 and the second classification value is 2.

[0181] In a specific embodiment, an SVM classifier is used to classify based on the fusion feature.

[0182] Among them, training the SVM classifier includes the following steps:

[0183] Step 1: Use the training data set to train the SVM classifier, where x i represents the feature vector of the i-th sample, and y i represents the classification value of the i-th sample.

[0184] Step 2: The SVM decision function. Specifically, the goal of the SVM is to find a hyperplane that maximizes its distance to the nearest sample point. Among them, for a linear SVM, the decision function can be expressed as:

[0185] f(x) = w T x + b

[0186] where w represents the weight vector; b represents the bias term.

[0187] Step 3: Kernel function. Specifically, for non-linear SVM, the data can be mapped to a high-dimensional space by introducing a kernel function (e.g., Radial Basis Function RBF), and the kernel function can be expressed as:

[0188] K(x i , x j ) = exp(-γ||x i - x j || 2 )

[0189] where γ represents the parameter of the kernel function.

[0190] In one embodiment, the touch test result includes a first test result; at this time, the fused features are analyzed to obtain the touch test result, specifically: the fused features are analyzed to obtain the test response result of the terminal device; and, the true response result of the terminal device is obtained; the test response result is compared with the true response result to determine whether the test response result of the terminal device is abnormal.

[0191] Specifically, define the true response result of the terminal device, which can be represented as a vector, as follows:

[0192] ExpectedResult = (r1, r1,..., r n )

[0193] where r i represents the i-th true response result of the touch test task.

[0194] Define the test response result of the terminal device, which can be represented as a vector, as follows:

[0195] ActualResult = (a1, a1,..., a n )

[0196] where a i represents the i-th test response result of the touch test task.

[0197] Result comparison: Compare the true response result with the test response result to determine whether they match, as follows:

[0198] Match = Compare(ExpectedResult, ActualResult)

[0199] where Compare represents the comparison function, which can be a simple element-by-element comparison or a more complex similarity metric.

[0200] Further, if the real response result matches the test response result, it indicates that the test response result of the terminal device is normal; otherwise, it indicates that the test response result of the terminal device is abnormal.

[0201] In a specific embodiment, the first test feature, the second test feature, and the third test feature are fused to obtain a fused feature, which can be specifically expressed as:

[0202] FusedData = f(ImageData, TextData, PixelData)

[0203] where f represents a fusion function, which can be a support vector machine (SVM), a random forest, or a deep learning model.

[0204] Please refer to Figure 11 , Figure 11 , which is a schematic structural diagram of an embodiment of the software testing device of the terminal device provided in this application. The software testing device 110 of the terminal device includes a determination module 111 and a testing module 112; the determination module 111 is used to determine the target movement path from the current position of the robotic arm to the target position; wherein, the target position is the position of the control to be tested in the terminal device; the testing module 112 is used to move the robotic arm to the target position according to the target movement path, so that the robotic arm performs a touch test task on the control to be tested.

[0205] Among them, the software testing device 110 of the terminal device further includes a verification module 113, and the verification module 113 is used to, after moving the robotic arm to the target position according to the target movement path so that the robotic arm performs a touch test task on the control to be tested, include: obtaining touch test data; analyzing the touch test data to obtain a touch test result.

[0206] Among them, the above touch test data includes at least one of the following: pixel data of the touch position of the robotic arm, test image data of the touch test process, task information of the touch test task; wherein, the touch test process includes the process of the robotic arm moving to the target position according to the target movement path; and / or, the above touch test result includes at least one of a first test result and a second test result, the first test result is used to characterize whether the test response result of the terminal device is abnormal, the test response result is the response situation of the terminal device based on the touch of the robotic arm on the control to be tested, and the second test result is used to characterize whether the robotic arm correctly performs the touch test task.

[0207] Among them, the above robotic arm includes an end effector for touching the control to be tested. The touch test data includes pixel data of the touch position of the robotic arm, test image data of the touch test process, and task information of the touch test task. The verification module 113 is used to analyze the touch test data to obtain a touch test result, including: determining the end effector and its corresponding first position and the control to be tested and its corresponding second position from the test image data of the touch test process as the first test data; and using the task information of the touch test task to determine the touch test type of the robotic arm as the second test data; and using the pixel data of the touch position of the robotic arm to determine the touch control actually touched by the end effector as the third test data; analyzing the first test data, the second test data, and the third test data to obtain the touch test result.

[0208] Among them, the verification module 113 is used to determine the end effector and its corresponding first position and the control to be tested and its corresponding second position from the test image data of the touch test process, including: performing object recognition on the test image data of the touch test process to obtain a number of first reference objects and their corresponding first reference positions; selecting the end effector and the control to be tested from the number of first reference objects, and using the first reference position of the first reference object as the end effector as the first position corresponding to the end effector, and using the first reference position of the first reference object as the control to be tested as the second position of the control to be tested.

[0209] Among them, the verification module 113 is used to select the end effector and the control to be tested from a number of first reference objects, including: selecting at least two first key objects from the number of first reference objects; classifying each first key object to determine the end effector and the control to be tested.

[0210] Among them, the verification module 113 is used to select at least two first key objects from a number of first reference objects, including: obtaining the key feature points of each first reference object; and obtaining the preset feature points of at least two preset objects in the database; for each first reference object, finding the preset feature points that match the key feature points of the first reference object to obtain the corresponding matching point pairs of the first reference object; selecting at least two first key objects from the number of first reference objects whose corresponding matching point pairs meet the matching requirements.

[0211] Among them, the verification module 113 is used to find out the preset feature points that match the key feature points of the first reference object to obtain the matching point pairs corresponding to the first reference object, including: obtaining the reference distance between the first feature descriptor of the key feature points of the first reference object and the second feature descriptors of each preset feature point; obtaining at least one preset feature point whose reference distance meets the first distance requirement as a candidate matching point; and selecting a candidate matching point and the key feature points of the first reference object to form the matching point pairs corresponding to the first reference object.

[0212] Among them, the steps for obtaining the first feature descriptor of the key feature points of the above-mentioned first reference object include: obtaining several object feature points of the first reference object; selecting key feature points from the several object feature points; and generating the first feature descriptor of the key feature points by using the object feature points located within the preset distance range of the key feature points.

[0213] Among them, the above-mentioned selection of key feature points from several object feature points includes: obtaining the response values of each object feature point; taking the object feature points whose response values meet the response requirements as key feature points; and / or the object feature points located within the preset distance range of the key feature points include: the object feature points other than the key feature points among the several object feature points.

[0214] Among them, the verification module 113 is used to determine the touch test type of the robotic arm by using the task information of the touch test task, including: extracting the task key information from the task information of the touch test task; and determining the touch test type of the robotic arm based on the task key information.

[0215] Among them, the verification module 113 is used to determine the touch control of the end effector by using the pixel data of the touch position of the robotic arm, including: performing object recognition on the test image data during the touch test process to obtain several second reference objects; and using the pixel data of the touch position of the robotic arm to select, from the several second reference objects, the second reference object that matches the touch position as the touch control of the end effector.

[0216] Among them, the pixel data of the touch position of the above-mentioned robotic arm includes the pixel points of the touch position of the robotic arm; the verification module 113 is used to determine the touch control of the end effector by using the pixel data of the touch position of the robotic arm, including: obtaining the key feature points of each second reference object; selecting the to-be-classified feature points that match the pixel points of the touch position from the key feature points of each second reference object; and classifying the second reference object corresponding to the to-be-classified feature points to determine the touch control.

[0217] Among them, the verification module 113 is used to analyze the first test data, the second test data, and the third test data to obtain a touch test result, including: extracting features from the first test data, the second test data, and the third test data respectively to obtain a first test feature, a second test feature, and a third test feature; fusing the first test feature, the second test feature, and the third test feature to obtain a fused feature; analyzing the fused feature to obtain a touch test result.

[0218] Among them, the above touch test result includes a second test result; the verification module 113 is used to analyze the fused feature to obtain a touch test result, including: classifying based on the fused feature to obtain a classification result; in response to the classification result being a first classification value, determining that the robotic arm correctly executes the touch test task; in response to the classification result being a second classification value, determining that the robotic arm does not correctly execute the touch test task.

[0219] Among them, the above touch test result includes a first test result; the verification module 113 is used to analyze the fused feature to obtain a touch test result, including: analyzing the fused feature to obtain a test response result of the terminal device; and obtaining a true response result of the terminal device; comparing the test response result with the true response result to determine whether the test response result of the terminal device is abnormal.

[0220] Among them, the determination module 111 is used to determine a target motion path from the current position of the robotic arm to the target position, including: obtaining an initial motion path from the current position of the robotic arm to the target position; performing path smoothing processing on the initial motion path to obtain a target motion path.

[0221] Please refer to Figure 12 , Figure 12 which is a schematic structural diagram of an embodiment of an electronic device provided by this application. The electronic device 120 includes a memory 121 and a processor 122 that are coupled to each other. The processor 122 is used to execute program instructions stored in the memory 121 to implement the steps of the software test method embodiment of any of the above terminal devices. In a specific implementation scenario, the electronic device 120 may include, but is not limited to: a microcomputer, a server. In addition, the electronic device 120 may also include mobile devices such as a laptop computer, a tablet computer, etc., which are not limited herein.

[0222] Specifically, the processor 122 is used to control itself and the memory 121 to implement the steps of any of the software testing method embodiments of the above terminal device. The processor 122 may also be referred to as a CPU (Central Processing Unit). The processor 122 may be an integrated circuit chip with signal processing capabilities. The processor 122 may also be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. Additionally, the processor 122 may be implemented jointly by integrated circuit chips.

[0223] Please refer to Figure 13 , Figure 13 FIG. is a schematic structural diagram of an embodiment of a computer-readable storage medium provided by the present application. The computer-readable storage medium 130 of the embodiment of the present application stores program instructions 131, and when the program instructions 131 are executed, the methods provided by any of the software testing method embodiments of the terminal device of the present application and any non-conflicting combinations are implemented. Among them, the program instructions 131 may form a program file and be stored in the above computer-readable storage medium 130 in the form of a software product, so that a computer device (which may be a personal computer, a server, or a network device, etc.) can execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned computer-readable storage medium 130 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc, or a terminal device such as a computer, a server, a mobile phone, or a tablet.

[0224] If the technical solution of this application involves personal information, before the product applying the technical solution of this application processes personal information, it has clearly informed the personal information processing rules and obtained the independent consent of the individual. If the technical solution of this application involves sensitive personal information, before the product applying the technical solution of this application processes sensitive personal information, it has obtained the individual's separate consent and at the same time meets the requirements of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If an individual voluntarily enters the collection scope, it is regarded as consenting to the collection of their personal information; or on the device for personal information processing, when the personal information processing rules are informed by obvious signs / information, personal authorization is obtained through pop-up messages or by asking the individual to upload their personal information by themselves, etc.; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0225] The above are only the implementation manners of this application, and do not limit the patent scope of this application accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of this application, or directly or indirectly applied in other related technical fields, shall be similarly included in the patent protection scope of this application.

Claims

1. A software testing method for a terminal device, characterized in that: The method comprises: Determine a target motion path from the current position of the robot arm to the target position; wherein the target position is the position of the control to be tested in the terminal device; The mechanical arm is used to move to the target position according to the target motion path, so that the mechanical arm performs a touch test task on the control to be tested.

2. The method according to claim 1, characterized in that After using the mechanical arm to move to the target position according to the target motion path so that the mechanical arm performs a touch test task on the control to be tested, the method further includes: Get touch test data; The touch test data is analyzed to obtain a touch test result.

3. The method according to claim 2, characterized in that The touch test data includes at least one of the following: pixel data of the touch position of the robotic arm, test image data of the touch test process, and task information of the touch test task; wherein the touch test process includes a process in which the robotic arm moves to the target position according to the target motion path; And / or, the touch test result includes at least one of a first test result and a second test result, the first test result is used to characterize whether a test response result of the terminal device is abnormal, the test response result is the response of the terminal device based on the robotic arm touching the control to be tested, and the second test result is used to characterize whether the robotic arm correctly performs the touch test task.

4. The method according to claim 3, characterized in that The mechanical arm includes an end effector for touching the control to be tested, the touch test data includes pixel data of the touch position of the mechanical arm, test image data of the touch test process and task information of the touch test task; the touch test data is analyzed to obtain a touch test result, including: Determining the end effector and its corresponding first position and the control to be tested and its corresponding second position from the test image data of the touch test process as first test data; and Determining the touch test type of the robot arm using the task information of the touch test task as second test data; and Determine the touch control actually touched by the end effector by using the pixel data of the touch position of the robot arm as the third test data; The first test data, the second test data and the third test data are analyzed to obtain the touch test result.

5. The method according to claim 4, characterized in that The step of determining the end effector and its corresponding first position and the control to be tested and its corresponding second position from the test image data of the touch test process includes: Performing object recognition on the test image data of the touch test process to obtain a plurality of first reference objects and corresponding first reference positions; From the several first reference objects, the end effector and the control to be tested are selected, and the first reference position of the first reference object of the end effector is used as the first position corresponding to the end effector, and the first reference position of the first reference object of the control to be tested is used as the second position of the control to be tested.

6. The method according to claim 5, characterized in that The step of selecting the end effector and the control to be tested from the plurality of first reference objects comprises: Selecting at least two first key objects from the plurality of first reference objects; Each of the first key objects is classified to determine the end effector and the control to be tested.

7. The method according to claim 6, characterized in that The step of selecting at least two first key objects from the plurality of first reference objects comprises: Acquire key feature points of each of the first reference objects; and, Acquire preset feature points of at least two preset objects in a database; For each of the first reference objects, searching for preset feature points that match key feature points of the first reference object to obtain matching point pairs corresponding to the first reference object; At least two first key objects whose corresponding matching point pairs meet the matching requirement are selected from the plurality of first reference objects.

8. The method according to claim 7, characterized in that The step of searching for preset feature points that match key feature points of the first reference object to obtain matching point pairs corresponding to the first reference object includes: Acquire a reference distance between a first feature descriptor of a key feature point of the first reference object and a second feature descriptor of each of the preset feature points; Acquire at least one preset feature point whose reference distance meets the first distance requirement as a candidate matching point; A candidate matching point and a key feature point of the first reference object are selected to form a matching point pair corresponding to the first reference object.

9. The method according to claim 8, characterized in that The step of acquiring the first feature descriptor of the key feature point of the first reference object comprises: Acquire a plurality of object feature points of the first reference object; Selecting the key feature point from the plurality of object feature points; A first feature descriptor of the key feature point is generated by using the object feature point located within a preset distance range of the key feature point.

10. The method according to claim 9, characterized in that The step of selecting the key feature point from the plurality of object feature points comprises: Obtaining a response value of each feature point of the object; Taking the object feature points whose response values ​​meet the response requirements as the key feature points; And / or, the object feature points located within a preset distance range of the key feature points include: object feature points other than the key feature points among the plurality of object feature points.

11. The method according to claim 4, characterized in that The step of using the task information of the touch test task to determine the touch test type of the robotic arm includes: Extracting key task information from the task information of the touch test task; Based on the mission critical information, a touch test type of the robotic arm is determined.

12. The method according to claim 4, characterized in that The method of using the pixel data of the touch position of the robot arm to determine the touch control of the end effector includes: Performing object recognition on the test image data of the touch test process to obtain a plurality of second reference objects; Using the pixel data of the touch position of the robot arm, a second reference object matching the touch position is selected from the plurality of second reference objects as a touch control of the end effector.

13. The method according to claim 12, characterized in that The pixel data of the touch position of the robotic arm includes pixel points of the touch position of the robotic arm; the pixel data of the touch position of the robotic arm is used to select a second reference object matching the touch position from the plurality of second reference objects as a touch control of the end effector, including: Acquire key feature points of each of the second reference objects; Selecting, from the key feature points of each of the second reference objects, feature points to be classified that match the pixel points at the touch position; Classify the second reference object corresponding to the feature point to be classified to determine the touch control.

14. The method according to claim 4, characterized in that The analyzing the first test data, the second test data, and the third test data to obtain the touch test result includes: Performing feature extraction on the first test data, the second test data, and the third test data respectively to obtain a first test feature, a second test feature, and a third test feature; fusing the first test feature, the second test feature and the third test feature to obtain a fused feature; The fusion feature is analyzed to obtain the touch test result.

15. The method according to claim 14, characterized in that The touch test result includes the second test result; and the analyzing the fusion feature to obtain the touch test result includes: Perform classification based on the fusion features to obtain a classification result; In response to the classification result being a first classification value, determining that the robotic arm correctly performs the touch test task; In response to the classification result being a second classification value, it is determined that the robot arm does not correctly perform the touch test task.

16. The method according to claim 14, characterized in that The touch test result includes the first test result; and the analyzing the fusion feature to obtain the touch test result includes: Analyzing the fusion feature to obtain a test response result of the terminal device; and, Obtaining a real response result of the terminal device; The test response result is compared with the actual response result to determine whether the test response result of the terminal device is abnormal.

17. The method according to claim 1, characterized in that Determining a target motion path from a current position of the robotic arm to a target position includes: Acquire an initial motion path from the current position of the robotic arm to the target position; Perform path smoothing processing on the initial motion path to obtain the target motion path.

18. A software testing device for a terminal device, characterized in that: The device comprises: A determination module, used to determine a target motion path from the current position of the robot arm to the target position; wherein the target position is the position of the control to be tested in the terminal device; The testing module is used to utilize the mechanical arm to move to the target position according to the target motion path, so that the mechanical arm performs a touch testing task on the control to be tested.

19. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory stores program instructions, and the processor is used to execute the program instructions to implement the test method of the terminal device according to any one of claims 1-17.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program instructions, and the program instructions can be executed to implement the terminal device testing method according to any one of claims 1-17.

Images