Mechanical arm control method and device, computer equipment and readable storage medium

Abstract

The invention relates to a mechanical arm control method and device, computer equipment and a readable storage medium. The method comprises the steps that for each reference point in three non-collinear reference points on a screen of first display equipment, under the condition that the tail end of a mechanical arm moves to the reference point, the three-dimensional coordinates of the tail end in a three-dimensional space where the mechanical arm is located are obtained and serve as the three-dimensional coordinates of the reference points; determining a coordinate conversion relation according to the respective three-dimensional coordinates of the three reference points; under the condition that a second relative position between the mechanical arm and the second display equipment is consistent with the first relative position, determining a to-be-positioned target screen coordinate in a screen space corresponding to the screen, and determining a three-dimensional coordinate of a pixel point at the target screen coordinate based on the coordinate conversion relation and the target screen coordinate; and controlling the tail end to move towards the pixel point at the target screen coordinate in the screen of the second display equipment based on the three-dimensional coordinate of the pixel point at the target screen coordinate. By adopting the method, the use scene can be widened.

Description

Technical Field

[0001] The present application relates to the field of robotic arm technology, and in particular to a robotic arm control method, device, computer equipment and readable storage medium. Background Art

[0002] With the development of science and technology, the application of robotic arms is becoming more and more extensive. For example, the robotic arms can be controlled to simulate the user's operations to operate the screen.

[0003] In traditional technology, a track-type robotic arm is usually used to simulate the user's operation to operate the screen. Specifically, the screen is first fixed in parallel with two mutually perpendicular double slide rails, and the horizontal axis movement is controlled by the two mutually perpendicular double slide rails. Then, a retractable robotic arm perpendicular to the double slide rails controls the up and down movement of the vertical axis to realize operations at various coordinate points on the screen.

[0004] However, the control of the rail-type robotic arm requires the slide rail and the plane position to be fixed and in a parallel position, which is not suitable for use in scenarios such as daily testing work and actual vehicle environments, limiting the usage scenarios. Summary of the invention

[0005] Based on this, it is necessary to provide a robot arm control method, device, computer equipment and readable storage medium that can broaden the usage scenarios in response to the above technical problems.

[0006] On the one hand, the present application provides a robotic arm control method, including: while maintaining a first relative position between a multi-degree-of-freedom robotic arm and a first display device of a target type, for each of three non-collinear reference points on the screen of the first display device, when the end of the robotic arm moves to the reference point, obtaining the three-dimensional coordinates of the end in the three-dimensional space where the robotic arm is located as the three-dimensional coordinates of the reference point; determining a coordinate transformation relationship based on the three-dimensional coordinates of each of the three reference points, the coordinate transformation relationship being used to determine the three-dimensional coordinates of a pixel point in the screen in the three-dimensional space; for a second display device of the target type, when the second relative position between the robotic arm and the second display device is consistent with the first relative position, determining the target screen coordinates to be positioned in the screen space corresponding to the screen, and determining the three-dimensional coordinates of the pixel point at the target screen coordinates based on the coordinate transformation relationship and the target screen coordinates; and controlling the end to move toward the pixel point at the target screen coordinates on the screen of the second display device based on the three-dimensional coordinates of the pixel point at the target screen coordinates.

[0007] On the other hand, the present application also provides a robotic arm control device, including: a three-dimensional coordinate acquisition module, which is used to maintain a first relative position between a multi-degree-of-freedom robotic arm and a first display device of the target type, and for each of three non-collinear reference points on the screen of the first display device, when the end of the robotic arm moves to the reference point, obtain the three-dimensional coordinates of the end in the three-dimensional space where the robotic arm is located as the three-dimensional coordinates of the reference point; a transformation relationship determination module, which is used to determine a coordinate transformation relationship based on the three-dimensional coordinates of each of the three reference points, and the coordinate transformation relationship is used to determine the three-dimensional coordinates of a pixel point in the screen in the three-dimensional space; a three-dimensional coordinate determination module, which is used to determine, for a second display device of the target type, when the second relative position between the robotic arm and the second display device is consistent with the first relative position, the target screen coordinates to be positioned in the screen space corresponding to the screen, and determine the three-dimensional coordinates of the pixel point at the target screen coordinate based on the coordinate transformation relationship and the target screen coordinates; a movement control module, which is used to control the end to move to the pixel point at the target screen coordinate in the screen of the second display device based on the three-dimensional coordinates of the pixel point at the target screen coordinate.

[0008] On the other hand, the present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above-mentioned robot arm control method when executing the computer program.

[0009] On the other hand, the present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the above-mentioned robot arm control method are implemented.

[0010] On the other hand, the present application also provides a computer program product, including a computer program, which implements the steps in the above-mentioned robot arm control method when executed by a processor.

[0011] The above-mentioned robot control method, device, computer equipment, computer-readable storage medium and computer program product, when the multi-degree-of-freedom robot maintains a first relative position with the first display device of the target type, for each of the three non-collinear reference points on the screen of the first display device, when the end of the robot moves to the reference point, obtain the three-dimensional coordinates of the end in the three-dimensional space where the robot is located as the three-dimensional coordinates of the reference point; determine the coordinate transformation relationship according to the three-dimensional coordinates of each of the three reference points, and the coordinate transformation relationship is used to determine the three-dimensional coordinates of the pixel point in the screen in the three-dimensional space; for the second display device of the target type, when the second relative position between the robot and the second display device is consistent with the first relative position, determine the target screen coordinates to be located in the screen space corresponding to the screen, and determine the three-dimensional coordinates of the pixel point at the target screen coordinate based on the coordinate transformation relationship and the target screen coordinate; based on the three-dimensional coordinates of the pixel point at the target screen coordinate, control the end to move to the pixel point at the target screen coordinate in the screen of the second display device. Compared with the track-type robot, it does not need to fix the slide rail and the screen position and is in a parallel position, and can be applied to any use scenario such as daily testing and real vehicle environment, which broadens the use scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0013] FIG1 is a diagram of an application environment of a robot arm control method in some embodiments;

[0014] FIG2 is a schematic flow chart of a method for controlling a robotic arm in some embodiments;

[0015] FIG3 is a schematic diagram showing a robotic arm in three-dimensional space in some embodiments;

[0016] FIG4 is a schematic diagram of a screen space in some embodiments;

[0017] FIG5 is a schematic diagram of a screen space and a plane where the screen is located in a three-dimensional space in some embodiments;

[0018] FIG6 is a schematic diagram of determining a coordinate transformation relationship in some embodiments;

[0019] FIG7 is a schematic diagram of a process from obtaining screen coordinates to controlling the movement of a robotic arm in some embodiments;

[0020] FIG8 is a block diagram of a mechanical arm control device in some embodiments;

[0021] FIG. 9 is a diagram of the internal structure of a computer device in some embodiments. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0023] The robot arm control method provided in the embodiment of the present application can be applied to the application environment as shown in Figure 1. The application environment includes a multi-degree-of-freedom robot arm 102, a display device 104 and a computer device 106. The degree of freedom can be determined by the number of servos in the robot arm 102. For example, if there are 6 servos in the robot arm 102, the robot arm 102 is a six-degree-of-freedom robot arm. The components of the robot arm 102 include a control device, a base, an arm, a joint, a servo or an end effector, etc., and of course, other devices such as sensors can also be included. The control device is responsible for receiving instructions, processing data and controlling the movement of the robot arm to achieve precise control of the robot arm. The base can also be called a base, which is a fixed part of the robot arm and provides stability and support. The arm is a plurality of rigid or flexible components connecting the base and the end effector, which is responsible for transmitting motion and force, and can be linear, curved or retractable, etc. The joint is a rotating or sliding component connecting the arm, allowing the robot arm to move flexibly in different directions. The end effector is the "hand" of the robot arm, which is used to complete specific tasks such as touch operations, which can be but not limited to click, slide or double-click operations. The robot arm 102 can have multiple servos, for example, 4, 6 or 8 servos, which can be installed inside or near the joint to drive the joint movement. The type of servo can be but not limited to DC servo, stepper servo or servo servo.

[0024] The display device 104 may be a device that provides a display function and is applied to any scene. There is a screen in the display device. The display device 104 may be, but is not limited to, a mobile phone, a tablet, a computer, a billboard, a television or a projector, a medical device, a vehicle-mounted navigator or a portable navigator, a display device used in a vehicle's smart cockpit, etc. For example, the display device 104 may be an IVI (In-Vehicle Infotainment) device. IVI is an important part of the intelligent and digital development of modern automobiles. The vehicle may be any vehicle, such as a passenger car, a bus, a truck or a tractor, a sedan, a sports utility vehicle, a truck or a bus, a fuel vehicle or a new energy vehicle, an unmanned vehicle, an automatic driving vehicle, a semi-automatic driving vehicle or a non-automatic driving vehicle. The computer device 106 may be, but is not limited to, a desktop computer, a laptop computer, a smart phone, and a tablet computer, etc.

[0025] Specifically, in the robot control method provided by the present application, the display device 104 is located within the range of motion of the robot. The computer device 106 can communicate with the robot 102 to control the movement of the robot 102. Before operating the display device 104 through the robot 102, the computer device can first determine the three-dimensional coordinates of the three reference points in the screen of the display device 104 in the three-dimensional space where the robot 102 is located. The three-dimensional coordinates of the reference points are the coordinates of the reference points in the three-dimensional space. According to the three-dimensional coordinates of the three reference points, a coordinate transformation relationship is determined, and the coordinate transformation relationship is used to determine the three-dimensional coordinates of the pixel point in the screen. Then, the computer device 106 can determine the target screen coordinates to be located, and based on the target screen coordinates and the coordinate transformation relationship, determine the three-dimensional coordinates of the pixel point at the target plane coordinates, and based on the three-dimensional coordinates of the pixel point at the target screen coordinates, control the end of the robot to move to the pixel point at the target screen coordinates in the screen of the display device 104.

[0026] It should be noted that the determined coordinate conversion relationship can be used not only for the robot 102 to operate the display device 104, but also for any display device (referred to as the second display device) of the same type as the display device 104 (referred to as the first display device). The same type can be the same display device; or, the same type can be: the shape of the display device is consistent, the position of the screen in the display device is consistent, and the shape and size of the screen in the display device are consistent. The size is represented by the resolution. Therefore, for the second display device of the same type as the first display device, when the relative position between the second display device and the robot 102 is consistent with the relative position between the first display device and the robot 102 in the process of determining the three-dimensional coordinates of the reference point, the computer device 106 can determine the target screen coordinates to be located, and determine the three-dimensional coordinates of the pixel point at the target plane coordinates based on the target screen coordinates and the coordinate conversion relationship, and control the end of the robot arm to move to the pixel point at the target screen coordinates in the screen of the display device 104 based on the three-dimensional coordinates of the pixel point at the target screen coordinates.

[0027] In some embodiments, as shown in FIG2 , a method for controlling a robot arm is provided. The method may be executed by a computer device or a control device, or may be executed by a computer device and a control device together. The method is described by taking the computer device 106 in FIG1 as an example, and includes the following steps 202 to 208. Among them:

[0028] Step 202, while maintaining a first relative position between the multi-degree-of-freedom robotic arm and a first display device of the target type, for each of the three non-collinear reference points on the screen of the first display device, when the end of the robotic arm moves to the reference point, obtain the three-dimensional coordinates of the end in the three-dimensional space where the robotic arm is located as the three-dimensional coordinates of the reference point.

[0029] The target type can be any type, for example, it can be the same display device produced in the same batch. The first relative position can be any relative position that satisfies the first display device to be located within the range of motion of the robot arm. The purpose of the first display device being located within the range of motion of the robot arm is that the end of the robot arm can contact the screen of the first display device to facilitate operation of the screen. The three-dimensional space is a three-dimensional coordinate system, which can be, but is not limited to, a Cartesian coordinate system. The screen is rectangular.

[0030] The three non-collinear reference points refer to the three reference points that cannot be connected to form a straight line. The three reference points may be any three points on the screen of the first display device, for example, any three vertices among the four vertices of the screen of the first display device, for example, the vertex at the upper right corner, the vertex at the upper left corner, and the vertex at the lower right corner of the screen.

[0031] The robot arm can be manually pulled to change its posture, so the end of the robot arm can be moved to the reference point by manually pulling.

[0032] In some embodiments, the computer device can display a control interface corresponding to the robotic arm, and display prompt information for obtaining the three-dimensional coordinates of the reference point in the control interface, for example, "Please manually drag and move the end of the robotic arm to the upper right corner of the screen to obtain the three-dimensional coordinates of the pixel point at the upper right corner", "Please manually drag and move the end of the robotic arm to the upper left corner of the screen to obtain the three-dimensional coordinates of the pixel point at the upper left corner", "Please manually drag and move the end of the robotic arm to the lower right corner of the screen to obtain the three-dimensional coordinates of the pixel point at the lower right corner".

[0033] In some embodiments, the control interface may display a coordinate determination control. For each reference point, when the end of the robotic arm is moved to the reference point, the computer device responds to a trigger operation for the coordinate determination control and sends a coordinate acquisition instruction to the control device of the robotic arm. The control device determines the three-dimensional coordinates of the end of the current robotic arm in response to the coordinate acquisition instruction and returns the three-dimensional coordinates of the end of the current robotic arm to the computer device. The computer device uses the three-dimensional coordinates returned by the control device as the three-dimensional coordinates of the reference point.

[0034] Of course, the control device can obtain the angles of each servo of the current manipulator in response to the coordinate acquisition instruction, that is, obtain the angle of each servo, and return the angles of each servo of the current manipulator to the computer device. The computer device can calculate the three-dimensional coordinates of the end of the current manipulator according to the angles of each servo, and use the calculated three-dimensional coordinates as the three-dimensional coordinates of the reference point.

[0035] In some embodiments, the computer device may display the three-dimensional coordinates of each reference point in the control interface, and may also display the three-dimensional space and the robotic arm in the three-dimensional space.

[0036] In some embodiments, the control device of the robotic arm can construct a three-dimensional space according to the initial posture of the robotic arm. Specifically, the control device can use the center of the base of the robotic arm in the initial posture as the origin, the straight line perpendicular to the base surface as the positive direction of the Z axis, the horizontal direction parallel to the base as the positive direction of the X axis, and the direction perpendicular to the X axis, the Z axis and to the right as the positive direction of the Y axis, and the right refers to the direction to the right of the user when the user is facing the robotic arm, to establish a Cartesian coordinate system, and obtain the three-dimensional space where the robotic arm is located, as shown in Figure 3, which shows the three-dimensional space where the robotic arm is located established in this way. Of course, other methods can also be used to establish a three-dimensional space. The six circles in the robotic arm in Figure 3 represent the six joints of the robotic arm. Step 204, determine the coordinate transformation relationship based on the three-dimensional coordinates of each of the three reference points, and the coordinate transformation relationship is used to determine the three-dimensional coordinates of the pixel points on the screen in the three-dimensional space.

[0037] Among them, the coordinate conversion relationship is used to determine the three-dimensional coordinates of the pixel point according to the screen coordinates of the pixel point, and the screen coordinates refer to the coordinates in the screen space. The screen space is determined according to the size of the screen. The size of the screen is represented by the resolution, which can be but not limited to 1920×1080, and the unit is pixel. The screen space is a screen rectangular coordinate system, and the origin of the screen rectangular coordinate system is the first vertex of the screen, the positive direction of the Y axis is the direction from the first vertex to the second vertex, and the positive direction of the X axis is the direction from the first vertex to the third vertex. For example, the first vertex is the vertex in the upper right corner of the screen, the second vertex is the vertex in the upper left corner of the screen, and the third vertex is the vertex in the lower right corner of the screen. As shown in Figure 4, a schematic diagram of a screen rectangular coordinate system is shown, where point P1 is the vertex in the upper left corner of the screen, P2 is the vertex in the upper right corner of the screen, P3 is the vertex in the lower right corner of the screen, and P4 is the vertex in the lower left corner of the screen. Vertices are also pixel points.

[0038] Since the pixels on the screen are evenly arranged, the screen coordinates of the pixels in the rectangular coordinate system of the screen represent the arrangement position of the pixels. For example, if the screen coordinates of the pixel are (i, j), it means that the pixel is the i-th pixel in the positive direction of the X-axis and the j-th pixel in the positive direction of the Y-axis. 0≤i≤M-1, 0≤j≤N-1, M represents the number of pixels in the X-axis direction, N represents the number of pixels in the positive direction of the Y-axis, and the screen coordinates of the origin are (0,0).

[0039] In some embodiments, the computer device can calculate the plane normal vector based on the three-dimensional coordinates of each of the three reference points, and then determine the target plane based on the three-dimensional coordinates of any point among the three reference points and the plane normal vector. For example, assuming that the three reference points are P1, P2, and P3, the three-dimensional coordinates of P1 are (x1, y1, z1), the three-dimensional coordinates of P2 are (x2, y2, z2), and the three-dimensional coordinates of P3 are (x3, y3, z3). , using P1 and the plane normal vector Determine the plane, then the target plane can be expressed as: .

[0040] In some embodiments, in order to calculate the plane normal vector, the computer device can determine two vectors based on the three-dimensional coordinates of each of the three reference points, and calculate the cross product of the two vectors to obtain the plane normal vector. For example, two vectors can be determined based on P1, P2 and P3, respectively. and , then the plane normal vector , , , .

[0041] In the following embodiment, since the target plane is a plane uniquely determined by three reference points, each pixel point in the screen is located in the target plane, thereby determining the coordinates of the pixel point in the target plane, and the three-dimensional coordinates of the pixel point can be obtained. The computer device can determine the coordinate transformation relationship for determining the three-dimensional coordinates of the pixel point on the target plane according to the screen coordinates of the pixel point based on the target plane, thereby converting the screen coordinates of the pixel point into three-dimensional coordinates through the coordinate transformation relationship to obtain the three-dimensional coordinates of the pixel point. As shown in Figure 5, Figure 5 (a) shows the target plane where the screen is located in the three-dimensional space, and Figure 5 (b) shows the arrangement of each pixel point in the rectangular coordinate system of the screen. The target plane can be obtained by mapping the screen into the three-dimensional space.

[0042] As shown in Figure 6, a flow chart is provided for moving the end to different reference points and determining the three-dimensional coordinates of the different reference points. The end can be moved to any three points P1 to P4 on the screen in sequence, and when moving to each point, the angle of each servo of the robot arm is determined, and then the three-dimensional coordinates of the point are determined according to the angle of each servo, and then the coordinate conversion relationship can be determined according to the three-dimensional coordinates of the three points.

[0043] Step 206, for the second display device of the target type, when the second relative position between the robotic arm and the second display device is consistent with the first relative position, determine the target screen coordinates to be positioned in the screen space corresponding to the screen, and determine the three-dimensional coordinates of the pixel point at the target screen coordinates based on the coordinate conversion relationship and the target screen coordinates.

[0044] Among them, the first display device and the second display device can be the same display device, or two different display devices of the same type. The second relative position refers to the relative position between the robot arm and the second display device. The first relative position refers to the relative position between the robot arm and the first display device in the process of determining the three-dimensional coordinates of the reference point. The target screen coordinates can be the screen coordinates of any pixel point in the screen space. The target screen coordinates are the spatial position to which the end needs to move. The target screen coordinates can be screen coordinates determined in any scene, and can be but are not limited to screen coordinates selected in the scene for testing the screen.

[0045] Specifically, the computer device can calculate the three-dimensional coordinates of the pixel point at the target screen coordinates based on the target screen coordinates and the coordinate conversion relationship. For example, if the target screen coordinates are (1, 2), the pixel point at the target screen coordinates is the second pixel point in the X-axis direction and the third pixel point in the Y-axis direction, so the three-dimensional coordinates of the pixel point at (1, 2) can be calculated.

[0046] Step 208 : Based on the three-dimensional coordinates of the pixel point at the target screen coordinates, control the terminal to move toward the pixel point at the target screen coordinates on the screen of the second display device.

[0047] Specifically, the computer device can send the three-dimensional coordinates of the selected pixel point to the control device of the robotic arm. The control device can determine the target angle corresponding to each servo of the robotic arm based on the three-dimensional coordinates. The target angles corresponding to different servos may be the same or different. When the angle of each servo is the corresponding target angle, the end of the robotic arm moves to the selected pixel point. Alternatively, the computer device can calculate the target angle corresponding to each servo based on the three-dimensional coordinates of the selected pixel point, and then send the target angle corresponding to each servo to the control device of the robotic arm. Thereby, the control device can adjust the angle of each servo to the corresponding target angle, thereby moving the end to the selected pixel point.

[0048] In some embodiments, the computer device can control the terminal to move toward the selected pixel point based on the three-dimensional coordinates of the selected pixel point, and move to a position above the selected pixel point and where the distance between the terminal and the pixel point is less than a preset distance threshold. The distance threshold can be set as needed.

[0049] In some embodiments, when the end of the robotic arm moves to a selected pixel point, or moves above the selected pixel point and the distance between the end and the pixel point is less than a preset distance threshold, the computer device may send a touch instruction to the robotic arm so that the robotic arm performs a touch operation corresponding to the touch instruction on the display device. The touch operation may be, but is not limited to, click, double-click, or slide operations.

[0050] The robotic arm control method provided in the present application can be applied to any use scenario that requires positioning the screen in the display device by a robotic arm, which can be but not limited to the field of automated testing of the screen, automated performance testing, and other scenarios that require high-precision positioning in the plane. For example, it can be used in the use scenario of testing the IVI screen. With the development of smart cockpit technology, more and more car manufacturers are beginning to regard smart cockpits as the core function of their products. In order to ensure the stability and reliability of the smart cockpit system, testers need to write a large number of automated test scripts to test the cockpit system. However, the method of simulating clicks through automated scripts cannot simulate the real operation of the user, and there is a possibility of missing tests. At the same time, in some scenarios without the USB (Universal Serial Bus) permission of the vehicle-mounted system, it is impossible to perform automated testing by simulating clicks through scripts. At this time, the robotic arm control method provided in the present application can be applied to simulate the operation of real users, and the physical click method of the robotic arm can be used to simulate the operation of real users.

[0051] In the above-mentioned robot control method, when the multi-degree-of-freedom robot maintains a first relative position with the first display device of the target type, for each of the three non-collinear reference points on the screen of the first display device, when the end of the robot moves to the reference point, the three-dimensional coordinates of the end in the three-dimensional space where the robot is located are obtained as the three-dimensional coordinates of the reference point; according to the three-dimensional coordinates of each of the three reference points, a coordinate transformation relationship is determined, and the coordinate transformation relationship is used to determine the three-dimensional coordinates of the pixel point in the screen in the three-dimensional space; for the second display device of the target type, when the second relative position between the robot and the second display device is consistent with the first relative position, the target screen coordinates to be located in the screen space corresponding to the screen are determined, and based on the coordinate transformation relationship and the target screen coordinates, the three-dimensional coordinates of the pixel point at the target screen coordinates are determined; based on the three-dimensional coordinates of the pixel point at the target screen coordinates, the end is controlled to move to the pixel point at the target screen coordinates in the screen of the second display device. Compared with the track-type robot, it does not need to fix the slide rail and the screen position and be in a parallel position, and can be applied to any usage scenario such as daily testing and real vehicle environment, which broadens the usage scenario.

[0052] In some embodiments, the types of the first display device and the second display device are target types, and the method further includes: determining a test script corresponding to the target type, the test script is used to test at least one user interface in the display device of the target type, the test script includes multiple screen coordinates, the test script instructs to locate each screen coordinate in turn, and triggers a touch instruction for the display device after positioning; determining the target screen coordinates to be located in the screen space corresponding to the screen, including: executing the test script, and in the process of executing the test script, determining the target screen coordinates to be located from the multiple screen coordinates according to the positioning order of the multiple screen coordinates.

[0053] Each of the multiple screen coordinates may be the screen coordinate of a pixel point selected from the screen area occupied by the user interface.

[0054] Specifically, for each of the multiple screen coordinates, in the process of executing the test script, each screen coordinate can be used as the target screen coordinate to be located in turn. After the target screen coordinate is determined each time, the three-dimensional coordinates of the pixel point at the target screen coordinate can be determined based on the coordinate conversion relationship and the target screen coordinate. Then, based on the three-dimensional coordinates of the pixel point at the target screen coordinate, the coordinates of the end of the mechanical arm are adjusted to the three-dimensional coordinates of the pixel point at the target screen coordinate, so that the end of the mechanical arm can be moved to the position of the pixel point at the target screen coordinate in the screen of the second display device. Then, the test script can also trigger the sending of a touch instruction to the control device of the mechanical arm, so that the control device performs a touch operation on the screen of the second display device. Thereby, the test of at least one user interface in the screen of the second display device is realized.

[0055] In some embodiments, the computer device can display a virtual model of a second display device and display a user interface on the screen of the virtual model. In response to focusing on any position in the user interface, the screen coordinates of the pixel point at that position can be displayed. The screen coordinates in the test script can be determined in this way.

[0056] In this embodiment, during the testing of the display device, the three-dimensional coordinates of the pixel point at the target screen coordinates are determined based on the coordinate conversion relationship and the target screen coordinates, and based on the three-dimensional coordinates of the pixel point at the target screen coordinates, the end is controlled to move to the pixel point at the target screen coordinates in the screen of the second display device, and after the end moves to the pixel point at the target screen coordinates in the screen of the second display device, a touch command is sent to the robotic arm, so that during the test process, high-precision positioning and control at the millimeter level can be achieved, thereby improving the accuracy of the test.

[0057] In some embodiments, the method also includes: when the second relative position is inconsistent with the first relative position, determining an updated coordinate transformation relationship; executing a test script, and in the process of executing the test script, determining the target screen coordinates to be positioned from the multiple screen coordinates according to the positioning order of the multiple screen coordinates, and based on the updated coordinate transformation relationship and the target screen coordinates, determining the three-dimensional coordinates of the pixel point at the target screen coordinates, and based on the three-dimensional coordinates of the pixel point at the target screen coordinates, controlling the terminal to move to the pixel point at the target screen coordinates in the screen of the second display device.

[0058] Specifically, since the coordinate transformation relationship is determined based on the three-dimensional coordinates of the three reference points when the first display device and the robotic arm maintain the first relative position, if the second relative position is inconsistent with the first relative position, the coordinate transformation relationship is not applicable to determine the three-dimensional coordinates under the second relative position. Therefore, the computer device can re-execute the step of determining the coordinate transformation relationship to obtain an updated coordinate transformation relationship.

[0059] It should be noted that the three reference points refer to points at designated positions on the screen, such as vertices. Therefore, the three reference points are present on the screen of each target type of display device, and the three reference points do not refer to points on the screen of a single display device. Therefore, in some embodiments, in order to determine the updated coordinate transformation relationship, for each of the three non-collinear reference points on the screen of the second display device, when the end of the robotic arm moves to the reference point, the three-dimensional coordinates of the end of the robotic arm are obtained as the new three-dimensional coordinates of the reference point, so that the updated coordinate transformation relationship can be determined based on the new three-dimensional coordinates of each of the three reference points.

[0060] In this embodiment, when the second relative position is inconsistent with the first relative position, the test script can be adapted only by updating the coordinate transformation relationship without making complex modifications to the test script, thereby reducing maintenance costs.

[0061] In some embodiments, the three reference points are pixel points at the three vertices of the screen, and a coordinate transformation relationship is determined based on the three-dimensional coordinates of each of the three reference points, including: determining a first vector pointing from the three-dimensional coordinates of a first reference point among the three reference points to the three-dimensional coordinates of a second reference point among the three reference points, wherein the first vector is parallel to the Y axis of the screen space; determining a second vector pointing from the three-dimensional coordinates of the first reference point to the three-dimensional coordinates of a third reference point among the three reference points, wherein the second vector is parallel to the X axis of the screen space; and obtaining a coordinate transformation relationship based on the three-dimensional coordinates of the first reference point, the first vector, and the second vector.

[0062] The size of the display device is represented by a resolution, which may be, but is not limited to, 1920×1080, where 1920 is the length of the screen, 1080 is the width of the screen, and the unit is pixel. The screen space is a screen rectangular coordinate system, and the origin of the screen rectangular coordinate system is the first vertex of the screen, the positive direction of the Y axis is the direction from the first vertex to the second vertex, and the positive direction of the X axis is the direction from the first vertex to the third vertex, for example, the first vertex is the vertex at the upper right corner of the screen, the second vertex is the vertex at the upper left corner of the screen, and the third vertex is the vertex at the lower right corner of the screen.

[0063] Since the pixels on the screen are evenly arranged, the screen coordinates of the pixels in the rectangular coordinate system of the screen represent the arrangement position of the pixels. For example, if the screen coordinates of the pixel are (i, j), it means that the pixel is the i-th pixel in the positive direction of the X-axis and the j-th pixel in the positive direction of the Y-axis. 0≤i≤M-1, 0≤j≤N-1, M represents the number of pixels in the X-axis direction, N represents the number of pixels in the positive direction of the Y-axis, and the screen coordinates of the origin are (0,0).

[0064] The first vector is perpendicular to the second vector. For example, the first reference point is the vertex P2 at the upper right corner of the screen, the second reference point is the vertex P1 at the upper left corner of the screen, and the third reference point is the vertex P3 at the lower right corner of the screen. The first vector is , the second vector is The coordinate transformation relationship may include the first vector, the second vector, and the three-dimensional coordinates of the first reference point. Since the first vector is parallel to the Y axis of the screen space, and the second vector is parallel to the X axis of the screen space, the length of the first vector is the length of the screen in the Y axis direction (i.e., the number of pixels), and the length of the second vector is the length of the screen in the X axis direction.

[0065] Specifically, the coordinate transformation relationship can be: P (x, y, z) = P0 (x, y, z) + U × first vector + V × second vector, 0≤U≤1, 0≤V≤1. U represents the weight of the first vector, V represents the weight of the second vector, P (x, y, z) represents the three-dimensional coordinates of any pixel on the screen, and P0 (x, y, z) represents the three-dimensional coordinates of the first reference point.

[0066] In this embodiment, a coordinate transformation relationship is obtained based on the three-dimensional coordinates of the first reference point, the first vector, and the second vector. The method for determining the coordinate transformation relationship is simple and efficient.

[0067] In some embodiments, based on the coordinate transformation relationship and the target screen coordinates, the three-dimensional coordinates of the pixel point at the target screen coordinates are determined, including: determining the weight of the first vector and the weight of the second vector according to the target screen coordinates and the size of the screen; performing weighted summation of the first vector and the second vector according to the weight of the first vector and the weight of the second vector to obtain a coordinate offset vector; and using the three-dimensional coordinates of the first reference point and the coordinate offset vector to determine the three-dimensional coordinates of the pixel point at the target screen coordinates.

[0068] Specifically, the computer device adds the three-dimensional coordinates of the first reference point and the coordinate offset vector to obtain the three-dimensional coordinates of the pixel point at the target screen coordinates.

[0069] In some embodiments, the computer device may subtract the ordinate of the target screen coordinate from the ordinate of the screen coordinate of the first reference point to obtain the ordinate difference, and use the ratio of the ordinate difference to the length of the screen in the Y-axis direction (i.e., the number of pixels in the Y-axis direction) as the weight of the first vector. Similarly, the computer device may subtract the abscissa of the screen coordinate of the first reference point from the abscissa of the target screen coordinate to obtain the abscissa difference, and use the ratio of the abscissa difference to the length of the screen in the X-axis direction (i.e., the number of pixels in the X-axis direction) as the weight of the second vector. Coordinate offset vector = weight of first vector × first vector + weight of second vector × second vector.

[0070] For example, the first reference point is the origin (0,0), the direction of the first vector is consistent with the positive direction of the Y axis of the screen rectangular coordinate system, the direction of the second vector is consistent with the positive direction of the X axis of the screen rectangular coordinate system, the length of the screen in the positive direction of the X axis is 1920 (pixels) and the length in the positive direction of the Y axis is 1080 (pixels), if the target screen coordinates are (640,540), then the weight of the first vector = (540-0) / 1080=1 / 2, the weight of the second vector = (640-0) / 1920=1 / 3. Therefore, the coordinates P (x, y, z) calculated according to the coordinate conversion relationship P (x, y, z) = P0 (x, y, z) + 1 / 2 × first vector + 1 / 3 × second vector are the three-dimensional coordinates of the pixel point at the target screen coordinates, where 1 / 2 × first vector + 1 / 3 × second vector is the coordinate offset vector.

[0071] As shown in FIG7 , a schematic diagram of calculating three-dimensional coordinates according to target screen coordinates and controlling the motion of the robotic arm according to the calculated three-dimensional coordinates is shown. First, the screen coordinates (x, y) are obtained, and then the weights of the first vector and the second vector are calculated according to the screen coordinates. The weights of the first vector and the second vector are then brought into the coordinate transformation relationship to calculate the three-dimensional coordinates. Next, the inverse kinematics algorithm of the robotic arm is used to calculate the target angle corresponding to each servo in the robotic arm, and then the angle of each servo is adjusted toward the corresponding target angle to control the motion of the robotic arm.

[0072] In this embodiment, the three-dimensional coordinates of the pixel point at the target screen coordinates are quickly and accurately calculated by weighting the first vector and the second vector and combining them with the three-dimensional coordinates of the first reference point.

[0073] In some embodiments, when the end of the robotic arm moves to a reference point, the three-dimensional coordinates of the end in the three-dimensional space where the robotic arm is located are obtained, including: when the end of the robotic arm moves to the reference point, obtaining the current servo angles of the robotic arm; and determining the three-dimensional coordinates of the end in the three-dimensional space based on the servo angles.

[0074] Specifically, the computer device can obtain the angles of each servo from the control device by sending instructions to the control device of the robotic arm.

[0075] In some embodiments, the control device of the robotic arm can calculate the three-dimensional coordinates of the end of the robotic arm according to the angles of each servo, and return the calculated three-dimensional coordinates to the computer device.

[0076] In this embodiment, the three-dimensional coordinates of the terminal in the three-dimensional space can be accurately and quickly determined according to the angles of each steering engine.

[0077] In some embodiments, based on the three-dimensional coordinates of the pixel point at the target screen coordinates, the control terminal moves to the pixel point at the target screen coordinates in the screen of the second display device, including: sending a control instruction carrying the three-dimensional coordinates of the pixel point at the target screen coordinates to the robotic arm.

[0078] Among them, the control instructions are used for the robotic arm to determine the target angle corresponding to each servo of the robotic arm according to the three-dimensional coordinates of the pixel point at the target screen coordinates, and adjust the angle of each servo to the corresponding target angle to control the end to move to the pixel point at the target screen coordinates in the screen of the second display device.

[0079] Specifically, the control device of the robotic arm can use the three-dimensional coordinates to calculate the angle that each servo of the robotic arm needs to reach when the end is located at the three-dimensional coordinates according to the inverse kinematics algorithm of the robotic arm.

[0080] In this embodiment, the movement of the robotic arm can be controlled by sending three-dimensional coordinates to the robotic arm, and because the three-dimensional coordinates of the pixel points are sent, the movement of the robotic arm can be controlled at the pixel level, and millimeter-level precision control can be achieved.

[0081] In some embodiments, as shown in Figure 8, a flow chart of a robotic arm control method is provided. First, a screen rectangular coordinate system and a robotic arm three-dimensional coordinate system are established, and the three-dimensional coordinates of three vertices in the screen in the three-dimensional coordinate system are collected. A coordinate transformation relationship is generated based on the three-dimensional coordinates of each of the three vertices collected. Then, the three-dimensional coordinates are obtained based on the screen coordinates and the coordinate transformation relationship. Then, the movement of each servo of the robotic arm is planned through inverse kinematics.

[0082] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0083] Based on the same inventive concept, the embodiment of the present application also provides a robot arm control device for implementing the robot arm control method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more robot arm control device embodiments provided below can refer to the limitations of the robot arm control method above, and will not be repeated here.

[0084] In some embodiments, as shown in FIG8 , a robot arm control device is provided, including: a three-dimensional coordinate acquisition module 802, a conversion relationship determination module 804, a three-dimensional coordinate determination module 806 and a movement control module 808, wherein:

[0085] The three-dimensional coordinate acquisition module 802 is used to maintain a first relative position between the multi-degree-of-freedom robotic arm and a first display device of the target type, and for each of the three non-collinear reference points on the screen of the first display device, when the end of the robotic arm moves to the reference point, obtain the three-dimensional coordinates of the end of the robotic arm in the three-dimensional space where the robotic arm is located, as the three-dimensional coordinates of the reference point.

[0086] The conversion relationship determination module 804 is used to determine the coordinate conversion relationship according to the three-dimensional coordinates of the three reference points. The coordinate conversion relationship is used to determine the three-dimensional coordinates of the pixel points on the screen in the three-dimensional space.

[0087] The three-dimensional coordinate determination module 806 is used to determine the target screen coordinates to be positioned in the screen space corresponding to the screen for the second display device of the target type, when the second relative position between the robotic arm and the second display device is consistent with the first relative position, and determine the three-dimensional coordinates of the pixel point at the target screen coordinates based on the coordinate conversion relationship and the target screen coordinates.

[0088] The movement control module 808 is used to control the terminal to move to the pixel point at the target screen coordinates on the screen of the second display device based on the three-dimensional coordinates of the pixel point at the target screen coordinates.

[0089] In some embodiments, the device also includes a script determination module, which is used to determine a test script corresponding to the target type. The test script is used to test at least one user interface in a display device of the target type. The test script includes multiple screen coordinates determined based on at least one user interface. The test script instructs to locate each screen coordinate in turn and trigger a touch instruction for the display device after positioning; the three-dimensional coordinate determination module 806 is also used to execute the test script, and in the process of executing the test script, determine the target screen coordinates to be located from the multiple screen coordinates according to the positioning order of the multiple screen coordinates; the device also includes an instruction sending module, which is used to send a touch instruction to the robotic arm after the end moves to the pixel point at the target screen coordinates in the screen of the second display device.

[0090] In some embodiments, the conversion relationship determination module 804 is also used to determine the updated coordinate conversion relationship when the second relative position is inconsistent with the first relative position; the device is also used to execute a test script, and in the process of executing the test script, determine the target screen coordinates to be located from the multiple screen coordinates according to the positioning order of the multiple screen coordinates, and determine the three-dimensional coordinates of the pixel point at the target screen coordinates based on the updated coordinate conversion relationship and the target screen coordinates, and based on the three-dimensional coordinates of the pixel point at the target screen coordinates, control the terminal to move to the pixel point at the target screen coordinates in the screen of the second display device.

[0091] In some embodiments, the three reference points are pixel points at the three vertices of the screen, and the transformation relationship determination module 804 is further used to determine a first vector pointing from the three-dimensional coordinates of a first reference point among the three reference points to the three-dimensional coordinates of a second reference point among the three reference points, and the first vector is parallel to the Y axis of the screen space; determine a second vector pointing from the three-dimensional coordinates of the first reference point to the three-dimensional coordinates of a third reference point among the three reference points, and the second vector is parallel to the X axis of the screen space; and obtain a coordinate transformation relationship based on the three-dimensional coordinates of the first reference point, the first vector, and the second vector.

[0092] In some embodiments, the three-dimensional coordinate determination module 806 is also used to determine the weight of the first vector and the weight of the second vector based on the target screen coordinates and the size of the screen; perform weighted summation of the first vector and the second vector based on the weight of the first vector and the weight of the second vector to obtain a coordinate offset vector; and use the three-dimensional coordinates of the first reference point and the coordinate offset vector to determine the three-dimensional coordinates of the pixel point at the target screen coordinates.

[0093] In some embodiments, the three-dimensional coordinate acquisition module 802 is also used to obtain the current servo angles of the robotic arm when the end of the robotic arm moves to the reference point; and determine the three-dimensional coordinates of the end in three-dimensional space based on the servo angles.

[0094] In some embodiments, the mobile control module 808 is also used to send a control instruction carrying the three-dimensional coordinates of the pixel point at the target screen coordinates to the robotic arm; wherein the control instruction is used for the robotic arm to determine the target angle corresponding to each servo of the robotic arm according to the three-dimensional coordinates of the pixel point at the target screen coordinates, and adjust the angle of each servo to the corresponding target angle, so as to control the end to move to the pixel point at the target screen coordinates in the screen of the second display device.

[0095] Each module in the above-mentioned robot arm control device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0096] In some embodiments, a computer device is provided, which may be a control device in a vehicle, and its internal structure diagram may be shown in FIG9. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for controlling a mechanical arm is implemented.

[0097] Those skilled in the art will understand that the structure shown in FIG. 9 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0098] In some embodiments, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0099] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0100] In some embodiments, a computer program product is provided, including a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0101] It should be noted that the data involved in this application (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0102] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0103] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0104] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A robot arm control method, characterized in that: The method comprises: In a case where a first relative position is maintained between a multi-degree-of-freedom robotic arm and a first display device of a target type, for each of three non-collinear reference points on a screen of the first display device, when a tip of the robotic arm moves to the reference point, obtaining a three-dimensional coordinate of the tip in the three-dimensional space where the robotic arm is located as the three-dimensional coordinate of the reference point; Determine a coordinate transformation relationship according to the three-dimensional coordinates of each of the three reference points, wherein the coordinate transformation relationship is used to determine the three-dimensional coordinates of the pixel points on the screen in the three-dimensional space; For a second display device of the target type, when a second relative position between the robotic arm and the second display device is consistent with the first relative position, determining the target screen coordinates to be positioned in the screen space corresponding to the screen, and determining the three-dimensional coordinates of the pixel point at the target screen coordinates based on the coordinate conversion relationship and the target screen coordinates; Based on the three-dimensional coordinates of the pixel point at the target screen coordinates, the terminal is controlled to move toward the pixel point at the target screen coordinates in the screen of the second display device.

2. The method according to claim 1, characterized in that: The method further comprises: Determine a test script corresponding to the target type, the test script is used to test at least one user interface in a display device of the target type, the test script includes a plurality of screen coordinates determined based on the at least one user interface, the test script instructs to locate each of the screen coordinates in turn, and triggers a touch instruction for the display device after the positioning; The determining the target screen coordinates to be located in the screen space corresponding to the screen includes: Executing the test script, and in the process of executing the test script, determining the target screen coordinates to be located from the multiple screen coordinates according to the positioning order of the multiple screen coordinates; The method further comprises: After the end moves to the pixel point at the target screen coordinates in the screen of the second display device, a touch instruction is sent to the mechanical arm.

3. The method according to claim 2, characterized in that The method further comprises: In the case where the second relative position is inconsistent with the first relative position, determining an updated coordinate transformation relationship; Execute the test script, and in the process of executing the test script, determine the target screen coordinates to be located from the multiple screen coordinates according to the positioning order of the multiple screen coordinates, and determine the three-dimensional coordinates of the pixel point at the target screen coordinates based on the updated coordinate conversion relationship and the target screen coordinates, and based on the three-dimensional coordinates of the pixel point at the target screen coordinates, control the terminal to move to the pixel point at the target screen coordinates in the screen of the second display device.

4. The method according to claim 1, characterized in that: The three reference points are pixel points at three vertices of the screen, and determining the coordinate transformation relationship according to the three-dimensional coordinates of each of the three reference points includes: Determine a first vector pointing from the three-dimensional coordinates of a first reference point among the three reference points to the three-dimensional coordinates of a second reference point among the three reference points, wherein the first vector is parallel to the Y axis of the screen space; Determine a second vector pointing from the three-dimensional coordinates of the first reference point to the three-dimensional coordinates of a third reference point among the three reference points, wherein the second vector is parallel to the X-axis of the screen space; A coordinate transformation relationship is obtained based on the three-dimensional coordinates of the first reference point, the first vector, and the second vector.

5. The method according to claim 4, characterized in that The determining the three-dimensional coordinates of the pixel point at the target screen coordinates based on the coordinate conversion relationship and the target screen coordinates includes: Determining a weight of the first vector and a weight of the second vector according to the target screen coordinates and the size of the screen; According to the weight of the first vector and the weight of the second vector, performing weighted summation on the first vector and the second vector to obtain a coordinate offset vector; The three-dimensional coordinates of the pixel point at the target screen coordinates are determined using the three-dimensional coordinates of the first reference point and the coordinate offset vector.

6. The method according to claim 1, characterized in that When the end of the mechanical arm moves to the reference point, obtaining the three-dimensional coordinates of the end in the three-dimensional space where the mechanical arm is located includes: When the end of the robotic arm moves to the reference point, obtaining the current angles of each servo of the robotic arm; According to the steering gear angles, the three-dimensional coordinates of the terminal in the three-dimensional space are determined.

7. The method according to claim 1, characterized in that The controlling the terminal to move toward the pixel point at the target screen coordinates on the screen of the second display device based on the three-dimensional coordinates of the pixel point at the target screen coordinates includes: Sending a control instruction carrying the three-dimensional coordinates of the pixel point at the target screen coordinates to the robotic arm; Among them, the control instruction is used for the robotic arm to determine the target angle corresponding to each servo of the robotic arm according to the three-dimensional coordinates of the pixel point at the target screen coordinates, and adjust the angle of each servo to the corresponding target angle to control the end to move to the pixel point at the target screen coordinates on the screen of the second display device.

8. A robot arm control device, characterized in that: The device comprises: a three-dimensional coordinate acquisition module, for obtaining, for each of three non-collinear reference points on the screen of the first display device, the three-dimensional coordinates of the end of the multi-degree-of-freedom robot arm in the three-dimensional space where the robot arm is located when the end of the robot arm moves to the reference point, as the three-dimensional coordinates of the reference point, while maintaining a first relative position between the multi-degree-of-freedom robot arm and a first display device of the target type; A conversion relationship determination module, used to determine a coordinate conversion relationship according to the three-dimensional coordinates of each of the three reference points, wherein the coordinate conversion relationship is used to determine the three-dimensional coordinates of the pixel point on the screen in the three-dimensional space; a three-dimensional coordinate determination module, for determining, for a second display device of the target type, target screen coordinates to be positioned in a screen space corresponding to the screen when a second relative position between the robotic arm and the second display device is consistent with the first relative position, and determining the three-dimensional coordinates of a pixel point at the target screen coordinates based on the coordinate conversion relationship and the target screen coordinates; A movement control module is used to control the terminal to move toward the pixel point at the target screen coordinates on the screen of the second display device based on the three-dimensional coordinates of the pixel point at the target screen coordinates.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images