Touch equipment control method and device, computer equipment and storage medium

Through the interface image analysis of touch device and the control of robotic arm, flexible and efficient automatic operation of touch device is achieved, solving the problems of cumbersome and high cost in the existing technology, and improving intelligence and robustness.

CN118981272AInactive Publication Date: 2024-11-19GONGDADI INNOVATION TECH SHENZHEN CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411429837.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-14
Publication Date
2024-11-19
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The prior art has problems such as cumbersome operation, high cost and low flexibility when controlling touch devices, especially affected by the update of operating system and application versions.

Method used

By acquiring the current interface image of the touch device, image analysis is performed to identify operable elements, and target touch operation is determined according to preset operation tasks, control the robotic arm to perform touch operations to complete the task, and obtain the interface feedback image after execution.

Benefits of technology

Improves the intelligent automation of touch devices, reduces the labor cost required to complete repetitive operation tasks, and enhances flexibility and robustness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981272A_ABST
    Figure CN118981272A_ABST
Patent Text Reader

Abstract

The invention relates to the field of artificial intelligence, and provides a touch equipment control method, device and equipment and a computer storage medium, and the method comprises the steps: obtaining a current image of an interface displayed by the touch equipment; performing image analysis on the current image of the interface to obtain at least one operable element in the current image of the interface; according to a preset operation task and the operable element, determining a target touch operation required for completing the operation task in the current image of the interface; and controlling a mechanical arm to execute the target touch operation on the touch equipment, and obtaining an interface feedback image displayed by the touch equipment after the target touch operation is executed. The current image of the interface is subjected to image analysis, so that the mechanical arm is controlled to conduct touch control to achieve the preset operation task, the intelligence of the mechanical arm for controlling the touch control equipment is improved, and the labor cost needed for completing the repeated operation task is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a touch device control method, device, computer equipment and storage medium. Background Art

[0002] With the development of technology, touch devices have become the mainstream electronic devices used in daily life. When operating touch devices to achieve some tasks, there are some repetitive operations. Manual completion is cumbersome and costly, and completion through scripts has certain limitations and low flexibility. It is easily affected by operating system and application version updates and application script detection. How to more flexibly and effectively control touch devices to automatically complete certain tasks has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of the present application is to provide a touch device control method, device, equipment and computer storage medium, aiming to improve the flexibility and intelligence of touch device automatic control.

[0004] In a first aspect, the present application provides a touch device control method, the touch device control method comprising the following steps: Obtaining a current image of the interface displayed by the touch device; Performing image analysis on the current image of the interface to obtain at least one operable element in the current image of the interface; Determine, according to a preset operation task and the operable elements, a target touch operation required in the current image of the interface to complete the operation task; The mechanical arm is controlled to perform the target touch operation on the touch device, and an interface feedback image displayed by the touch device after the target touch operation is performed is acquired.

[0005] In a second aspect, the present application further provides a touch device control device, the touch device control device comprising: An image acquisition module, used to acquire a current image of the interface displayed by the touch control device; An image analysis module, used to perform image analysis on the current image of the interface to obtain at least one operable element in the current image of the interface; A touch operation determination module, used to determine a target touch operation required in the current image of the interface to complete the operation task according to a preset operation task and the operable element; The touch operation execution module is used to control the mechanical arm to execute the target touch operation on the touch device, and obtain an interface feedback image displayed by the touch device after executing the target touch operation.

[0006] In a third aspect, the present application further provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the touch device control method as described above is implemented.

[0007] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the touch device control method as described above is implemented.

[0008] The present application provides a touch device control method, apparatus, device and computer storage medium, which obtains the current image of the interface displayed by the touch device; performs image analysis on the current image of the interface to obtain at least one operable element in the current image of the interface; determines the target touch operation required in the current image of the interface to complete the operation task according to the preset operation task and the operable element; controls the mechanical arm to perform the target touch operation on the touch device, and obtains the interface feedback image displayed by the touch device after performing the target touch operation. Since the image analysis is performed on the current image of the interface to control the mechanical arm to perform touch to achieve the preset operation task, the intelligence of the mechanical arm controlling the touch device is improved, and the manpower cost required to complete repetitive operation tasks is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 A schematic diagram of a flow chart of a touch device control method provided in one embodiment of the present application; Figure 2 A schematic diagram of a sub-step flow chart of performing image analysis on a current image of an interface provided in an embodiment of the present application; Figure 3 A schematic block diagram of a touch device control apparatus provided in an embodiment of the present application; Figure 4 The present invention is a block diagram showing the structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0012] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0013] Embodiments of the present application provide a touch device control method, apparatus, computer equipment, and computer-readable storage medium.

[0014] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0015] Please refer to Figure 1 , Figure 1 A schematic flow chart of a touch device control method provided in an embodiment of the present application. The touch device control method can be used in a robotic arm to control the touch device through the robotic arm to complete a preset operation task.

[0016] like Figure 1 As shown, the touch device control method includes steps S101 to S104.

[0017] Step S101: Acquire the current image of the interface displayed by the touch device.

[0018] For example, in order to control the robot arm to operate the touch device, it is first necessary to perceive the content displayed by the touch device by acquiring the current image of the interface of the touch device. The controller for controlling the robot arm can be connected to the device for acquiring the current image of the interface in a communication manner, such as wirelessly connecting via WiFi, Bluetooth, etc., or preferentially connecting via a data transmission line, which is not limited here.

[0019] In some implementations, acquiring a current image of an interface displayed by the touch-control device includes: Acquire a current image of the interface obtained by photographing the touch device with a photographing device; or, The current image of the interface is obtained by performing a screen capture operation on the touch control device.

[0020] Exemplarily, a camera may be disposed at a position relative to the display screen of the touch device to capture content displayed on the touch device, thereby obtaining a current image of the interface. The controller of the robotic arm obtains the current image of the interface through a communication connection with the camera.

[0021] For example, the controller of the robot arm can also communicate with the touch device to obtain the current image of the interface obtained by the touch device performing a screen capture operation. The screen capture operation performed by the touch device can be an image obtained by screenshot or a video frame in a video stream obtained by screen recording, which is not limited here.

[0022] Step S102: performing image analysis on the current image of the interface to obtain at least one operable element in the current image of the interface.

[0023] For example, the display screen of the touch device can display different interfaces of various applications, such as the homepage, comment page, user homepage, private message page, etc. of a social application. The page layouts of pages from different applications or different pages of the same application may be quite different, but the functions available to users of each application are similar, such as liking, commenting, sending private messages, etc. Based on this feature, the operable elements in the current image of the interface can be determined by performing image analysis on the current image of the interface, so as to automatically control the robotic arm to complete the preset operation task by performing certain operations.

[0024] In some implementations, step S102 performs image analysis on the current image of the interface to obtain at least one operable element in the current image of the interface, including: Step S1021: matching a preset graphic in the current image of the interface based on a preset visual matching model to obtain at least one touchable element in the current image of the interface; or, Step S1022: Recognize the current image of the interface based on a preset visual recognition model to determine at least one touchable element in the current image of the interface; or, Step S1023: input the current image of the interface into a preset intelligent agent, and determine at least one touchable element in the current image of the interface based on the intelligent agent.

[0025] For example, in user interface design, graphical expressions are usually used to quickly convey specific operations or functions to users, so that users from different languages ​​and cultural backgrounds can understand. For example, in the user interfaces of various applications, the "Like" function is usually represented by a heart-shaped icon or a thumbs-up icon, the "Collection" function is represented by a star-shaped icon or a bookmark icon, the "Share" function is represented by a curved arrow or a row of dots, the "Private Message" function is represented by an envelope icon, the "Search" function is represented by a magnifying glass icon, and the "Return" function is represented by "←" or "×". Therefore, the visual matching model can determine the touchable elements in the current image of the interface by matching the preset graphics in the current image of the interface. For example, if an operation task of liking a certain content is pre-set, this operation task can be completed by matching a touchable element with a heart-shaped or thumbs-up outline on the page of the content.

[0026] Specifically, the elements in the current image of the interface can be pixelated to convert the elements from high-pixel blocks to low-pixel blocks, and then the overlap between the pixelated elements and the preset graphics is calculated. If the overlap is greater than a preset value, the element is determined to be a touchable element.

[0027] Exemplarily, a specific interface can also be identified by training a visual recognition model. By collecting interface images of different pages in an application and annotating the operable elements in the interface images as training samples for training the visual recognition model, a visual recognition model capable of identifying operable elements in a specific interface can be trained, such as identifying operable elements in live broadcast interfaces, application pop-up windows, and other interfaces. The visual recognition model can be, for example, a convolutional neural network (CNN), but is of course not limited thereto and is not limited here.

[0028] Exemplarily, the touchable elements in the current image of the interface can also be identified by an agent, wherein the agent can be implemented based on a large model, such as GPT, Claude, etc. Specifically, the agent can read the current image of the interface and understand the content displayed in the current image of the interface, thereby determining the touchable elements in the current image of the interface and the corresponding meanings of the touchable elements.

[0029] Please refer to Figure 2 , Figure 2 A schematic diagram of the sub-step flow of performing image analysis on the current image of the interface provided in one embodiment of the present application.

[0030] like Figure 2As shown, since the visual matching model requires fewer operating resources, the visual recognition model requires the second most operating resources, and the intelligent agent requires the most operating resources, after acquiring the current image of the interface, the touchable elements are determined preferentially through step S1021. When the visual matching model cannot determine the touchable elements required to complete the operation task, step S1022 is executed. When the visual recognition model is also unable to determine the touchable elements required to complete the operation task, step S1023 is executed to call the intelligent agent for image analysis, so as to avoid waste of operating resources, reduce the cost of image analysis and improve analysis efficiency.

[0031] Of course, this is not limited to this, and the present application does not limit the execution order between step S1021-step S1023.

[0032] Step S103: determining, according to a preset operation task and the operable elements, a target touch operation required in the current image of the interface to complete the operation task.

[0033] Exemplarily, compared with the related art of completing operation tasks based on preset scripts or templates, the present solution can receive a variety of operation tasks, and based on the operable elements identified in the current image of the interface in step S102, determine the target touch operation that needs to be performed in the current image of the interface to complete the operation task, thereby expanding the application scenarios of the touch device control method.

[0034] In some implementations, determining, based on the preset operation task and the operable element, a target touch operation required in the current image of the interface to complete the operation task includes: Based on a preset intelligent agent, determining an operation subtask in the current image of the interface according to the operation task; The operation position, operation type and operation content of the target touch operation are determined according to the operation subtask and the operable element, wherein the operation type includes at least one of clicking, long pressing, dragging, sliding and zooming.

[0035] Exemplarily, the operation task can be input into the agent in the form of natural language.

[0036] Exemplarily, the intelligent agent decomposes the implementation process of the operation task into multiple operation subtasks, and determines what operation the robot arm needs to perform in the interface displayed by the current image of the interface to achieve the operation task. For example, if the operation task is to add a user as a friend, the robot arm first needs to open a specific application in the touch device, find the option to add a friend in the application, and enter the identity information for adding the friend.

[0037] Exemplarily, after determining the operation subtask, the robot arm needs to perform the operation content based on the specific operation type at the corresponding operation position, where the operation type includes click, long press, drag, slide, zoom, etc. For example, assuming that the operation task is to send a message to a specific friend, the robot arm needs to perform a sliding operation in the interface, and the operation content is to match a specific user name; after entering the message sending page, click the text box to invoke the input method for input, and the operation content is to enter the content to be sent.

[0038] Step S104: controlling the robot arm to perform the target touch operation on the touch device, and obtaining an interface feedback image displayed by the touch device after performing the target touch operation.

[0039] Exemplarily, in order to determine that the robotic arm successfully executes the target touch operation, it is also necessary to obtain the interface feedback image displayed by the touch device after executing the target touch operation. Specifically, after the robotic arm feeds back the information that the target touch operation is completed, the content displayed by the touch device is obtained through a shooting device or a screen capture operation.

[0040] In some implementations, the controlling mechanical arm performs the target touch operation on the touch device, and obtains an interface feedback image displayed by the touch device after performing the target touch operation, including: When the interface feedback image does not conform to the expected image, the robotic arm is controlled to perform an interface return operation and / or re-execute the target touch operation.

[0041] Exemplarily, the interface feedback image is judged by an intelligent agent. If the interface feedback image does not meet expectations after executing the target touch operation, it may be due to an operation error that causes the wrong interface to be entered, or it may be due to an operation failure that causes the original interface to remain.

[0042] Exemplarily, in the case of an operation error, a return operation needs to be performed to return to the previous interface in order to re-execute the target touch operation; in the case of an operation failure, the robotic arm is directly controlled to perform the target touch operation again.

[0043] Exemplarily, the effectiveness and accuracy of the target touch operation is ensured through the interface feedback image, and possible errors are adjusted in time, thereby ensuring the robustness of the touch device control method executed by the robotic arm.

[0044] In some implementations, the controlling mechanical arm performs the target touch operation on the touch device, and obtains an interface feedback image displayed by the touch device after performing the target touch operation, including: According to the operation position, a movement parameter of the mechanical arm for performing the target touch operation is determined based on a preset mapping function.

[0045] For example, the touch screens of most touch devices are capacitive touch screens, which determine the location of the touch by detecting the transfer of charge. In order to enable the robotic arm to interact with the touch screen of the touch device, the fingertips of the robotic arm are made of conductive material so that the touch screen can recognize the touch through the transfer of charge between the fingertips and the screen.

[0046] Exemplarily, after the operating position of the robotic arm is determined, it is necessary to control the robotic arm to touch the operating position. The operating position is usually expressed in the form of pixel coordinates in the current image of the interface. Therefore, the movement parameters of the robotic arm can be determined according to the operating position, so that the robotic arm moves to the operating position for operation. It can be understood that there is a certain mapping relationship between the operating position and the target position of the movement of the robotic arm. Therefore, the movement parameters of the robotic arm can be determined according to the operating position through a preset mapping function.

[0047] In some embodiments, before determining the movement parameters of the mechanical arm for performing the target touch operation based on the operation position and a preset mapping function, the method further includes: Acquire a first coordinate range corresponding to the image coordinate system of the current image of the interface and a second coordinate range corresponding to the mechanical arm coordinate system of the mechanical arm; Determining a mapping parameter of the mapping function according to a ratio of the first coordinate range to the second coordinate range; The step of determining the movement parameters of the mechanical arm for performing the target touch operation based on a preset mapping function according to the operation position further includes: The second coordinate of the operating position in the robot arm coordinate system is determined according to the ratio of the first coordinate of the operating position in the image coordinate system to the movement parameter.

[0048] Exemplarily, the motion coordinate system of the robotic arm is taken as the robotic arm coordinate system, and the motion range of the robotic arm is the first coordinate range of the robotic arm coordinate system, for example, it can be (100, 200); the pixel coordinate system in the current image of the interface is taken as the image coordinate system, and the vertex coordinates of the current image of the interface are the second coordinate range of the image coordinate system, for example, it can be (1125, 2436).

[0049] Exemplarily, before controlling the movement of the robotic arm, the position of the robotic arm in the robotic arm coordinate system is located at the origin of the image coordinate system, so that the origins of the robotic arm coordinate system and the image coordinate system coincide with each other. In this way, the mapping parameters between the robotic arm coordinate system and the image coordinate system can be determined by the ratio between the first coordinate range and the second coordinate range, so that the second coordinate of the operating position in the robotic arm coordinate system can be determined as the movement parameter of the robotic arm according to the ratio of the first coordinate of the operating position in the image coordinate system to the movement parameter.

[0050] For example, if the first coordinate range of the image coordinate system is (1125, 2436) and the second coordinate range of the robot coordinate system is (100, 200), the mapping parameter between the two is (11.25, 24.36). Assuming that the first coordinate of the operation position in the image coordinate system is (X, Y), the second coordinate of the target position of the robot movement in the robot coordinate system is (X', Y') = (X÷11.25, Y÷24.36).

[0051] The touch device control method provided in the above embodiment obtains the current image of the interface displayed by the touch device; performs image analysis on the current image of the interface to obtain at least one operable element in the current image of the interface; determines the target touch operation required in the current image of the interface to complete the operation task according to the preset operation task and the operable element; controls the mechanical arm to perform the target touch operation on the touch device, and obtains the interface feedback image displayed by the touch device after performing the target touch operation. By performing image analysis on the current image of the interface to control the mechanical arm to perform touch to achieve the preset operation task, the intelligence of the mechanical arm controlling the touch device is improved, and the labor cost required to complete repetitive operation tasks is reduced.

[0052] See also Figure 3 , Figure 3 It is a schematic block diagram of a touch device control apparatus provided in an embodiment of the present application. The touch device control apparatus can be configured in a server or a terminal to execute the aforementioned touch device control method.

[0053] like Figure 3 As shown, the touch device control apparatus includes: an image acquisition module 110 , an image analysis module 120 , a touch operation determination module 130 , and a touch operation execution module 140 .

[0054] An image acquisition module 110 is used to acquire a current image of the interface displayed by the touch device; An image analysis module 120, configured to perform image analysis on the current image of the interface to obtain at least one operable element in the current image of the interface; A touch operation determination module 130, configured to determine, according to a preset operation task and the operable elements, a target touch operation required in the current image of the interface to complete the operation task; The touch operation execution module 140 is used to control the robot arm to execute the target touch operation on the touch device, and obtain an interface feedback image displayed by the touch device after executing the target touch operation.

[0055] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0056] The method and apparatus of the present application can be used in many general or special computing system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0057] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in Figure 4 Runs on the computer device shown.

[0058] See also Figure 4 , Figure 4 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be a server or a terminal.

[0059] like Figure 4 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a storage medium and an internal memory.

[0060] The storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any touch device control method.

[0061] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0062] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any touch device control method.

[0063] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0064] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0065] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps: Obtaining a current image of the interface displayed by the touch device; Performing image analysis on the current image of the interface to obtain at least one operable element in the current image of the interface; Determine, according to a preset operation task and the operable elements, a target touch operation required in the current image of the interface to complete the operation task; The mechanical arm is controlled to perform the target touch operation on the touch device, and an interface feedback image displayed by the touch device after the target touch operation is performed is acquired.

[0066] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the touch device control described above can refer to the corresponding process in the aforementioned touch device control method embodiment, which will not be repeated here.

[0067] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the touch device control method of the present application.

[0068] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0069] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0070] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0071] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A touch device control method, characterized in that: The method comprises: Obtaining a current image of the interface displayed by the touch device; Performing image analysis on the current image of the interface to obtain at least one operable element in the current image of the interface; Determine, according to a preset operation task and the operable elements, a target touch operation required in the current image of the interface to complete the operation task; The mechanical arm is controlled to perform the target touch operation on the touch device, and an interface feedback image displayed by the touch device after the target touch operation is performed is acquired.

2. The touch device control method according to claim 1, characterized in that: The acquiring the current image of the interface displayed by the touch control device includes: Acquire a current image of the interface obtained by photographing the touch device with a photographing device; or, The current image of the interface is obtained by performing a screen capture operation on the touch control device.

3. The touch device control method according to claim 1, characterized in that: The performing image analysis on the current image of the interface to obtain at least one operable element in the current image of the interface includes: Matching a preset graphic in the current image of the interface based on a preset visual matching model to obtain at least one touchable element in the current image of the interface; or, Recognize the current image of the interface based on a preset visual recognition model to determine at least one touchable element in the current image of the interface; or, The current image of the interface is input into a preset intelligent agent, and at least one touchable element is determined in the current image of the interface based on the intelligent agent.

4. The touch device control method according to claim 3, characterized in that: The determining, based on the preset operation task and the operable element, a target touch operation required in the current image of the interface to complete the operation task includes: Based on a preset intelligent agent, determining an operation subtask in the current image of the interface according to the operation task; The operation position, operation type and operation content of the target touch operation are determined according to the operation subtask and the operable element, wherein the operation type includes at least one of clicking, long pressing, dragging, sliding and zooming.

5. The touch device control method according to claim 4, characterized in that: The controlling mechanical arm performs the target touch operation on the touch device, and obtains an interface feedback image displayed by the touch device after performing the target touch operation, including: According to the operation position, a movement parameter of the mechanical arm for performing the target touch operation is determined based on a preset mapping function.

6. The touch device control method according to claim 5, characterized in that: Before determining the movement parameters of the mechanical arm for performing the target touch operation based on the operation position and a preset mapping function, the method further includes: Acquire a first coordinate range corresponding to the image coordinate system of the current image of the interface and a second coordinate range corresponding to the mechanical arm coordinate system of the mechanical arm; Determining a mapping parameter of the mapping function according to a ratio of the first coordinate range to the second coordinate range; The step of determining the movement parameters of the mechanical arm for performing the target touch operation based on a preset mapping function according to the operation position further includes: The second coordinate of the operating position in the robot arm coordinate system is determined according to the ratio of the first coordinate of the operating position in the image coordinate system to the movement parameter.

7. The touch device control method according to any one of claims 1 to 6, characterized in that: The controlling mechanical arm performs the target touch operation on the touch device, and obtains an interface feedback image displayed by the touch device after performing the target touch operation, including: When the interface feedback image does not conform to the expected image, the robotic arm is controlled to perform an interface return operation and / or re-execute the target touch operation.

8. A touch device control device, characterized in that: The touch device control device comprises: An image acquisition module, used to acquire a current image of the interface displayed by the touch control device; An image analysis module, used to perform image analysis on the current image of the interface to obtain at least one operable element in the current image of the interface; A touch operation determination module, used to determine a target touch operation required in the current image of the interface to complete the operation task according to a preset operation task and the operable element; The touch operation execution module is used to control the mechanical arm to execute the target touch operation on the touch device, and obtain an interface feedback image displayed by the touch device after executing the target touch operation.

9. A computer device, characterized in that: The computer device comprises a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the touch device control method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the touch device control method according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Software testing method and device based on image recognition, equipment and medium

    CN113934630A

  • Touch screen testing method, device and system and storage medium

    CN114546740A

  • Automatic control method and device of instant messaging software, electronic equipment and medium

    CN117729168A

  • Control system, control method, and program

    JP2020160562A

  • Method of programming an industrial robot

    US20190202058A1