Manipulator, system and method for automatic operation of mouse

Abstract

The invention relates to the technical field of automation, in particular to a manipulator, system and method for automatic operation of a mouse, the manipulator comprises a mechanical arm and a collection part arranged on the mechanical arm, and the collection part is used for collecting information and sending the collected information to an upper computer; the mechanical palm is connected with the mechanical arm; the mechanical fingers are connected with the palm; the driving component and the electric control component are arranged in the mechanical palm, the driving component drives any mechanical finger in the multiple mechanical fingers to execute target mouse operation, the electric control component controls the driving component to execute automatic mouse operation according to a mouse operation instruction of the upper computer, and the upper computer generates the mouse operation instruction according to collected information. Therefore, the problems of high installation complexity, poor compatibility and convenience and the like caused by the fact that interaction between the manipulator and the mouse depends on a software level and the software dependency is too high in the prior art are solved.

Description

Technical Field

[0001] The present application relates to the field of automation technology, and in particular to a manipulator, system and method for automatic operation of a mouse. Background Art

[0002] With the rapid development of artificial intelligence, robotics, and human-computer interaction, the application of manipulators has expanded from traditional industrial manufacturing to more complex and delicate tasks in daily life, such as automated office and remote operation. The control technology of manipulators has also evolved from traditional physical buttons or remote controls to more complex voice control and visual recognition systems. These technologies make the interaction between users and manipulators more natural and intuitive, greatly improving the applicability and flexibility of manipulators.

[0003] At present, the interaction technology between the robot and the computer mouse mainly relies on the software level, which means that the user must pre-install specific software or drivers on the controlled computer to realize the control function of the robot. This dependence will lead to the following problems:

[0004] 1. Software dependency: (1) Installation and configuration complexity: Users need to have certain technical knowledge to install and configure the necessary software. This not only increases the difficulty of use, but may also cause errors during the installation process, affecting the function of the robot. (2) Compatibility issues: The software may require specific adjustments for different operating systems or computer configurations. This customized solution limits the versatility and flexibility of the robot. 2. Lack of plug-and-play function: Robot control systems often do not have true plug-and-play function, that is, users cannot simply connect the robot to any computer and start using it immediately, but need to perform a series of settings and configurations, which has poor flexibility. Summary of the invention

[0005] The present application provides a manipulator, system and method for automatic mouse operation to solve the problems in the related art that the interaction between the manipulator and the mouse depends on the software level, the software dependence is too high, resulting in high installation complexity, poor compatibility, versatility and flexibility.

[0006] The first aspect of the present application provides a robot for automatic mouse operation, comprising the following steps: a robot arm and a collection component arranged on the robot arm, wherein the collection component is used to collect information and send the collected information to a host computer; a robot palm connected to the robot arm and multiple robot fingers connected to the palm; a driving component and an electric control component arranged in the robot palm, wherein the driving component drives any robot finger among the multiple robot fingers to perform a target mouse operation, and the electric control component controls the driving component to perform the automatic mouse operation according to the mouse operation instruction of the host computer, wherein the host computer generates the mouse operation instruction according to the collected information.

[0007] Optionally, the acquisition component includes at least one of a visual sensor and a sound sensor.

[0008] Optionally, the drive component includes at least one of a linear motor and a rotary motor.

[0009] Optionally, the electronic control component includes a communication module and a processor, wherein the communication module communicates with a host computer, and the processor is used to receive mouse operation instructions sent by the host computer through the communication module, and control the drive component to perform mouse automation operations according to the mouse operation instructions of the host computer.

[0010] Optionally, the electronic control component also includes a power supply for supplying power to the manipulator.

[0011] Optionally, the robotic finger is a rigid robotic finger, a soft robotic finger, or a rigid-flexible coupled robotic finger.

[0012] A second aspect of the present application provides a system for automatic mouse operation, comprising: a manipulator for automatic mouse operation according to the above embodiment; and a host computer, wherein the host computer communicates with the manipulator to control the manipulator to perform automatic mouse operation.

[0013] A third aspect of the present application provides a method for mouse automation operation, wherein the method performs mouse automation operation based on a manipulator for mouse automation operation in the above-mentioned embodiment, wherein the method comprises the following steps: obtaining acquisition information of an acquisition component of the manipulator; identifying the screen image and the mouse position in the acquisition information; locating the cursor position and the target point position of the mouse in the screen image, wherein the target point position is determined according to the user's operation requirements; generating a mouse operation instruction according to the mouse position, the cursor position and the target point position, and controlling the manipulator to perform mouse automation operation based on the mouse operation instruction.

[0014] Optionally, a mouse operation instruction is generated based on the mouse position, cursor position and target point position, including: establishing a mapping relationship between the mouse position on the desktop and the cursor position on the screen; taking the difference between the positions of the cursor and the target point on the screen as input, and taking the adjustment amount of the position of the end of the robotic arm joint on the desktop as output, inputting the mouse position, cursor position and target point position into a PID algorithm, and generating a target moving distance of the end of the robotic arm joint, a target moving distance of the cursor and a click operation timing according to the PID algorithm and the mapping relationship; generating a mouse operation instruction based on the target moving distance of the end of the robotic arm joint, a target moving distance of the cursor and a click operation timing.

[0015] Optionally, before determining the position of the target point according to the user's operation requirements, the method further includes: identifying a voice control signal or a text control signal of the user; and obtaining the user's operation requirements based on the voice control signal or the text control signal.

[0016] Therefore, this application has at least the following beneficial effects:

[0017] The embodiment of the present application constructs a manipulator for automatic mouse operation, in which a driving component and an electric control component are integrated inside the manipulator, and the automatic operation of the manipulator on the mouse is realized based on the driving component and the electric control component, and no specific software or driver is pre-installed on the controlled computer, and plug-and-play can be realized, which improves the convenience of using the manipulator, and the electric control component can directly control the driving component to perform the automatic mouse operation according to the mouse operation instruction of the host computer, and the driving component directly controls the manipulator finger to perform the target mouse operation, which reduces the dependence on software, and further avoids the compatibility problem of software installation with the computer operating system, thereby improving the versatility and flexibility of the manipulator. Thus, the technical problems that the interaction between the manipulator and the mouse depends on the software level, the software dependence is too high, resulting in high installation complexity, poor compatibility, versatility and flexibility are solved.

[0018] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0020] Figure 1 A schematic diagram of the structure of a manipulator for automatic mouse operation provided according to an embodiment of the present application;

[0021] Figure 2 A hardware structure diagram of a manipulator for automatic mouse operation provided according to an embodiment of the present application;

[0022] Figure 3 A schematic diagram of a system for automatic mouse operation according to an embodiment of the present application;

[0023] Figure 4 A specific control diagram of a system for mouse automation operation provided according to an embodiment of the present application;

[0024] Figure 5 A flowchart of a method for mouse automation operation provided according to an embodiment of the present application;

[0025] Figure 6 A flowchart of mouse automated game operation provided according to an embodiment of the present application;

[0026] Figure 7A schematic diagram of a human controlling a manipulator based on a voice signal according to an embodiment of the present application;

[0027] Figure 8 The present invention is a schematic diagram of a process of human controlling a robot based on voice signals according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0029] The following describes the manipulator, system and method for mouse automation operation of the embodiment of the present application with reference to the accompanying drawings. In view of the problem that the current manipulator and computer mouse interaction technology mentioned in the above background technology mainly relies on the implementation at the software level, the software dependence is too high, resulting in high installation complexity, poor compatibility and convenience, the present application provides a manipulator for mouse automation operation, in which a driving component and an electric control component are integrated inside the manipulator, and the manipulator realizes the automatic operation of the mouse based on the driving component and the electric control component, and there is no need to pre-install specific software or driver on the controlled computer, which can realize plug-and-play, improve the convenience of using the manipulator, and the electric control component can directly control the driving component to perform the mouse automation operation according to the mouse operation instruction of the host computer, and the driving component directly controls the manipulator finger to perform the target mouse operation, reducing the dependence on the software. Thus, the problem that the interaction between the manipulator and the mouse in the technology depends on the software level, the software dependence is too high, resulting in high installation complexity, poor compatibility, versatility and flexibility is solved.

[0030] Specifically, Figure 1 A schematic diagram of the structure of a manipulator for automated mouse operation provided in an embodiment of the present application.

[0031] like Figure 1 As shown, the manipulator 10 for automatic mouse operation includes: a manipulator arm 11, a collection component 12, a manipulator palm 13, a plurality of manipulator fingers 14, a driving component 15 and an electric control component 16.

[0032] Among them, the collection component 12 is arranged on the robotic arm, the collection component is used to collect information and send the collected information to the host computer, the robotic palm 12 is connected to the robotic arm 11, and multiple robotic fingers 14 are connected to the robotic palm 12; the driving component 15 and the electronic control component 16 are both arranged in the robotic palm 12, the driving component 15 is used to drive any finger among the multiple robotic fingers 14 to perform the target mouse operation, and the electronic control component 16 controls the driving component 15 to perform the mouse automation operation according to the mouse operation instruction of the host computer, wherein the host computer generates the mouse operation instruction according to the collected information.

[0033] Among them, the multiple robotic fingers 11 can be five robotic fingers, the first to fifth fingers, and each finger performs different target mouse operations. For example, the first finger and the fifth finger are used to grasp the mouse, the second finger and the fourth finger are used to click the left and right buttons of the mouse respectively to complete single-shot, double-click, long press and other mouse operations, and the third finger is used to click the middle mouse button or scroll the wheel; the robotic arm can be any robotic arm that can meet the requirements of desktop mouse movement, without specific limitation, such as SCARA robotic arms and other robotic arms with fewer degrees of freedom.

[0034] It can be understood that the embodiment of the present application constructs a manipulator 10 for automatic mouse operation, in which a drive component and an electronic control component are integrated inside the manipulator 10, and the automatic operation of the mouse by the manipulator 10 is implemented based on the drive component 15 and the electronic control component 16. There is no need to pre-install specific software or drivers on the controlled computer, and plug-and-play can be implemented, thereby improving the convenience of using the manipulator. The electronic control component 16 can directly control the drive component 15 to perform automatic mouse operation according to the mouse operation instructions of the host computer, and the drive component 15 directly controls the manipulator finger 14 to perform the target mouse operation, which reduces the dependence on software and further avoids the compatibility issues of software installation with the computer operating system, thereby improving the versatility and flexibility of the manipulator.

[0035] In addition, it should be noted that the manipulator used for automated mouse operation is not limited to the manipulator in the embodiment of the present application, and various manipulators that can be used to operate a computer mouse fall within this scope.

[0036] In the embodiment of the present application, the acquisition component 12 includes at least one of a visual sensor and a sound sensor.

[0037] The visual sensor may be a camera, etc., the sound sensor may be a microphone, and the visual sensor and the sound sensor may also be integrated into one device, such as a hand-held camera with a microphone.

[0038] It can be understood that the collection component 12 provided on the robot arm 11 in the embodiment of the present application can be used to collect information on the screen of a computer connected to the robot arm, and the sound sensor can collect control instructions input by user voice, etc.

[0039] It should be noted that the visual information and auditory information collected by the collection component of the embodiment of the present application as the input of the host computer is not limited to visual sensors and sound sensors, but AR and VR information can also be used as the input of the host computer.

[0040] In the embodiment of the present application, the driving component 15 includes at least one of a linear motor and a rotary motor.

[0041] Among them, the number of linear motors and rotary motors can be set according to specific circumstances, for example, there are 3 linear motors and 1 rotary motor. The linear motor and rotary motor can control the linear motion and rotary motion of the robotic finger respectively.

[0042] It can be understood that the driving component 15 of the embodiment of the present application includes at least one of a linear motor and a rotary motor, and preferably both a linear motor and a rotary motor are provided to ensure precise control of the mechanical finger.

[0043] In the embodiment of the present application, the electronic control component 16 includes: a communication module and a processor.

[0044] The communication module communicates with the host computer, and the processor is used to receive the mouse operation instruction sent by the host computer through the communication module, and control the driving component 15 to perform the mouse automation operation according to the mouse operation instruction of the host computer.

[0045] It can be understood that the electric component 16 of the embodiment of the present application includes a communication module and a processor. The communication module is used to control the manipulator to communicate with the host computer, specifically, wireless communication via Bluetooth. The processor receives the mouse operation instructions sent by the host computer through the communication module, and controls the drive component 15 to perform mouse automation operations according to the mouse operation instructions of the host computer. The processor can be a control board, such as STM32F407.

[0046] In the embodiment of the present application, the electric control component 16 also includes a power supply for supplying power to the manipulator.

[0047] The power source may be a battery.

[0048] In the embodiment of the present application, the robot finger 14 is a rigid robot finger, a soft robot finger, or a rigid-flexible coupled robot finger.

[0049] It is understandable that the robotic finger 14 of the embodiment of the present application may be a rigid robotic finger, a soft robotic finger, or a rigid-flexible coupled robotic finger to improve the application scope of the robotic arm.

[0050] The following describes a manipulator for automatic mouse operation in an embodiment of the present application through a specific embodiment. The specific hardware structure is as follows: Figure 2 shown.

[0051] The robot is mainly composed of two parts: fingers and palm, which contains five fingers in total. Fingers 1 and 5 are used to grasp the mouse and can adapt to different sizes of mice. Fingers 2 and 4 are used for the left and right buttons of the motor mouse, which can complete mouse operations such as single click, double click, and long press. Fingers 3 is used for the middle button of the motor mouse or scrolling the wheel.

[0052] The palm contains the driving components and electronic control components of the robot, mainly including 3 linear motors and 1 rotary motor. The Bluetooth module is used for wireless communication, the control board (equivalent to the above-mentioned processor) is used to control the movement of the robot, and the battery is used to provide energy.

[0053] In summary, the manipulator of the embodiment of the present application can be plug-and-play, so that it can be seamlessly connected to computers of various brands and models without the need for users to perform complex software installation and configuration. In addition, the manipulator control system can run on a variety of operating systems, whether it is Windows, MacOS or Linux, to ensure stability and compatibility. This greatly reduces the time and technical barriers for users in setting up equipment and improves the user experience. And the plug-and-play feature makes the manipulator easier to be accepted and used by different groups of people, especially for people with disabilities and elderly users, who can more easily control computers through voice or simple operations.

[0054] According to the manipulator for automated mouse operation proposed in the embodiment of the present application, a manipulator for automated mouse operation is constructed, in which a drive component and an electronic control component are integrated inside the manipulator, and automated mouse operation of the manipulator is realized based on the drive component and the electronic control component. There is no need to pre-install specific software or drivers on the controlled computer, and plug-and-play can be realized, thereby improving the convenience of using the manipulator. The electronic control component can directly control the drive component to perform automated mouse operation according to the mouse operation instructions of the host computer, and the drive component directly controls the manipulator fingers to perform the target mouse operation, thereby reducing the dependence on software and further avoiding the compatibility issues of software installation with the computer operating system, thereby improving the versatility and flexibility of the manipulator.

[0055] The embodiment of the present application also provides a system for automatic mouse operation.

[0056] like Figure 3 As shown, the system 20 for automatic mouse operation includes: the above-mentioned manipulator 10 for automatic mouse operation and a host computer 21.

[0057] The host computer 21 communicates with the manipulator 10 to control the manipulator 10 to perform mouse automation operations.

[0058] It can be understood that the embodiments of the present application can control the manipulator to perform automated operations through a host computer to achieve the control function of the manipulator, wherein the host computer can communicate with the manipulator via Bluetooth.

[0059] Specifically, the specific control diagram of the system for mouse automation operation in the embodiment of the present application is as follows: Figure 4 The host computer receives the collected information (such as visual signals and voice signals) from the collection component, and controls the manipulator via Bluetooth.

[0060] Next, the method for automatic mouse operation proposed according to the embodiment of the present application is described with reference to the accompanying drawings.

[0061] Figure 5 It is a flow chart of a method for mouse automation operation according to an embodiment of the present application.

[0062] like Figure 5 As shown, the method for mouse automation operation, based on the above-mentioned manipulator for mouse automation operation, performs mouse automation operation, comprising the following steps:

[0063] In step S101, acquisition information of a acquisition component of a robot is acquired.

[0064] The collected information may be obtained by the above-mentioned collecting components, including computer screen information, user voice commands, and the like.

[0065] In step S102, the screen image and the mouse position in the collected information are identified.

[0066] The screen information specifically includes screen image and mouse information.

[0067] In step S103, the cursor position of the mouse and the target point position in the screen are located, wherein the target point position is determined according to the operation requirements of the user.

[0068] The position of the target point may be determined based on the user's operational requirements, for example, the position of the target point may be a target in a game, or may be the position of an icon on a screen.

[0069] Among them, the specific method of positioning can be selected according to the specific situation. Taking the positioning target as an example, the screen image is perspective transformed to obtain a 1920×1080 frame, and the predefined target pattern is convolved with the frame. The point with the highest convolution value is the position of the corresponding icon, thereby obtaining the position of the cursor and the target.

[0070] In an embodiment of the present application, before determining the target point position according to the user's operation requirements, it also includes: identifying the user's voice control signal or text control signal; and obtaining the user's operation requirements based on the voice control signal or text control signal.

[0071] It can be understood that the embodiments of the present application can recognize the user's voice control signal or text control signal, and then obtain the user's operation requirements based on the voice control signal or text control signal, thereby improving the flexibility and user-friendliness of the operation.

[0072] Specifically, the embodiment of the present application can obtain the user's voice control instructions through a microphone, and further convert and analyze the voice control signal into a text control signal, or directly set the text information through the host computer to generate a text control signal.

[0073] In step S104, a mouse operation instruction is generated according to the mouse position, the cursor position and the target point position, and the manipulator is controlled to perform the mouse automation operation based on the mouse operation instruction.

[0074] It can be understood that the embodiments of the present application can generate mouse operation instructions according to the mouse position, cursor position and target point position, and control the manipulator to perform mouse automation operations based on the mouse operation instructions.

[0075] In an embodiment of the present application, a mouse operation instruction is generated based on the mouse position, the cursor position and the target point position, including: establishing a mapping relationship between the mouse position on the desktop and the cursor position on the screen; taking the difference between the positions of the cursor and the target point on the screen as input, and taking the adjustment amount of the position of the end of the robotic arm joint on the desktop as output, inputting the mouse position, the cursor position and the target point position into a PID algorithm, and generating a target moving distance of the end of the robotic arm joint, a target moving distance of the cursor and a click operation timing according to the PID algorithm and the mapping relationship; generating a mouse operation instruction based on the target moving distance of the end of the robotic arm joint, a target moving distance of the cursor and a click operation timing.

[0076] It can be understood that the embodiments of the present application can establish a mapping relationship between the position of the mouse on the desktop and the position of the cursor on the screen to reduce the number of planning times of the robotic arm, ensure that the cursor can move to the target position quickly and accurately, take the difference between the position of the cursor and the target point on the screen as input, take the adjustment amount of the position of the end of the robotic arm joint on the desktop as output, input the mouse position, the cursor position and the target point position into the PID algorithm, and generate the target moving distance of the end of the robotic arm joint, the target moving distance of the cursor and the click operation timing according to the PID algorithm and the mapping relationship, and then generate the mouse operation instruction according to the target moving distance of the end of the robotic arm joint, the target moving distance of the cursor and the click operation timing to realize the operation of the target point.

[0077] Specifically, the process of the PID algorithm is: take the difference between the position of the cursor and the target point on the frame as input, and the adjustment amount of the position of the end of the robotic arm joint on the desktop as output; when the cursor does not move to the target point, control the end of the robotic arm joint to move a specific distance; when it is detected that the two overlap, execute a click operation.

[0078] Among them, the mapping relationship is constructed to establish a mapping relationship between the position of the cursor on the screen and the position of the mouse on the desktop, to control the moving speed of the end of the robotic arm joint. The distance the cursor moves on the screen is proportional to the distance the mouse moves on the desktop. The ratio of the two can be measured experimentally, and thus the mapping relationship between the two is established: y=ax, and a PID algorithm is established based on the mapping relationship. The deviation between the observed cursor position and the target position on the screen is used as input, and the proportional coefficient of the mapping relationship is used as the proportional term in the PID algorithm. The calculated output is the distance the mouse needs to move (the distance the end of the robotic arm joint needs to move).

[0079] It should be noted that the aforementioned explanations and descriptions of the embodiment of the manipulator and system for automatic mouse operation are also applicable to the method for automatic mouse operation of this embodiment, and will not be repeated here.

[0080] The method for automatic mouse operation is described below through specific embodiments.

[0081] Example 1: Automated operation based on visual signals. Taking the mouse automated game operation as an example, the target point is the target in the game. The specific process is as follows: Figure 6 As shown, including:

[0082] 1. Initialization.

[0083] The system deployed with cameras, manipulators and robotic arms is in any random position in the initial state. When the overall system receives instructions from the control host computer, it is initialized. The initialization mainly controls the robotic arm to move to a specific predefined posture.

[0084] 2. Find the position of the mouse on the two-dimensional plane.

[0085] Start the camera and find the position of the mouse on the two-dimensional plane, that is, the position of the mouse on the desktop.

[0086] 3. After image processing, control the robotic arm to grab the mouse.

[0087] After image processing, the robot arm is controlled to grab the mouse and adjusted to the optimal grabbing position.

[0088] 4. Adjust the position of the robot so that the camera on the back of the hand is aimed at the computer screen connected to the mouse.

[0089] 5. Start the target shooting game.

[0090] 6. Extract the edge of the computer screen.

[0091] A visual algorithm was developed based on OpenCV. First, the edge of the image information captured by the camera was extracted to determine whether there is a computer screen in the image and determine the location of the computer screen.

[0092] 7. Perspective transformation results in a two-dimensional screen.

[0093] If the computer screen is in the frame, the screen image is perspective transformed to obtain a 1920×1080 frame.

[0094] 8. Convolution positioning cursor and target.

[0095] The predefined pattern is convolved with the frame, and the point with the highest convolution value is the position of the corresponding icon, thereby obtaining the position of the cursor and the target.

[0096] 9. Control the robotic arm to move the cursor close to the target.

[0097] 10. Determine whether the cursor overlaps with the target.

[0098] 11. If they overlap, control the robot arm to complete the click operation, otherwise control the robot arm to move the cursor closer to the target.

[0099] 12. Repeat the above process until the host computer sends a control command to end the game.

[0100] Embodiment 2: Control of a manipulator for automatic mouse operation based on a voice signal, such as Figure 7 shown.

[0101] In the embodiment of the present application, a real-time speech recognition function is introduced, and the system can detect the instructions issued by the operator and convert them into text information. Basic functional functions such as controlling the movement of the end of the robot arm and controlling the operation of the robot are pre-defined, and relevant texts are written to introduce the task background and function functions. These application programming interfaces (APIs), background descriptions, and commands given by the operator will constitute prompts passed to ChatGPT. The large language model ChatGPT then sorts out and disassembles the logical relationship of the task.

[0102] Example 3: The execution process of controlling the robot to open the browser, close the browser and shut down the computer based on the voice signal, such as Figure 8 shown.

[0103] Taking a voice control task as an example, ChatGPT demonstrated its ability to understand user commands and generate corresponding codes. The instructions given by the operator were to open the browser on the desktop, close the browser, and turn off the computer power.

[0104] 1. Open the browser: ChatGPT first receives the command "open the browser on the desktop" and identifies the key action word "open" and the object "browser" through natural language processing technology.

[0105] Task decomposition: ChatGPT decomposes the task into the following two steps:

[0106] 1. Move the cursor: Use visual servo control technology to identify the location of the browser icon on the desktop, and use the control algorithm to calculate the cursor path and move it to the icon.

[0107] 2. Open the application: Use the double-click command to activate and open the browser.

[0108] Function call and execution: Call predefined APIs such as move_to("Internet") and double_click_left_button() to complete the operation. At the same time, the system will monitor the operation status in real time to ensure accurate execution of the task.

[0109] 2. Close the browser: Input "Close the browser" through voice, and ChatGPT will parse "close" and "browser".

[0110] Task decomposition: broken down into the following steps:

[0111] 1. Locate the close button: Move the cursor to the upper right corner of the screen.

[0112] 2. Execute the close operation: move the cursor to the close button and click the left mouse button.

[0113] 3. Turn off the computer power.

[0114] Task breakdown: This instruction is broken down into three steps:

[0115] 1. Launch the Start menu: Move the mouse to the Start button in the lower left corner of the screen.

[0116] 2. Navigate to Power Options: Locate the power button in the Start menu.

[0117] 3. Execute the shutdown operation: Select "Shutdown" in the power options and confirm the operation.

[0118] According to the method for mouse automation operation proposed in the embodiment of the present application, the screen image and mouse position in the information collected by the manipulator can be identified, and the mouse position and target point position in the screen image can be located. The user can determine the position of the target point according to simple operation requirements, generate mouse operation instructions according to the mouse position, cursor position and target point position, and then control the manipulator to perform mouse automation operation based on the mouse operation instructions. The degree of automation is high and the user does not need to perform complicated operations.

[0119] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0120] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0121] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0122] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0123] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

Claims

1. A manipulator for automatic operation of a mouse, characterized in that: include: A mechanical arm and a collection component arranged on the mechanical arm, wherein the collection component is used to collect information and send the collected information to a host computer; A mechanical palm connected to the mechanical arm and a plurality of mechanical fingers connected to the palm; A driving component and an electric control component are arranged in the palm of the robot, wherein the driving component drives any mechanical finger among the multiple mechanical fingers to perform a target mouse operation, and the electric control component controls the driving component to perform a mouse automation operation according to a mouse operation instruction of the host computer, wherein the host computer generates the mouse operation instruction according to the collected information.

2. The manipulator for automatic mouse operation according to claim 1, characterized in that: The acquisition component includes at least one of a visual sensor and a sound sensor.

3. The manipulator for automatic mouse operation according to claim 1, characterized in that: The driving component includes at least one of a linear motor and a rotary motor.

4. The manipulator for automatic mouse operation according to claim 1, characterized in that: The electronic control component includes a communication module and a processor, wherein the communication module communicates with the host computer, and the processor is used to receive mouse operation instructions sent by the host computer through the communication module, and control the drive component to perform mouse automation operations according to the mouse operation instructions of the host computer.

5. The manipulator for automatic mouse operation according to claim 4, characterized in that: The electric control component also includes a power supply for supplying power to the manipulator.

6. The manipulator for automatic mouse operation according to claim 1, characterized in that: The mechanical finger is a rigid mechanical finger, a soft mechanical finger or a rigid-flexible coupling mechanical finger.

7. A system for automatic mouse operation, characterized in that: include: The manipulator for automatic mouse operation according to any one of claims 1 to 6; A host computer, wherein the host computer communicates with the manipulator to control the manipulator to perform mouse automation operations.

8. A method for automatic mouse operation, characterized in that: The method performs mouse automation operation based on the manipulator for mouse automation operation according to any one of claims 1 to 6, wherein the method comprises the following steps: Obtaining collection information of the collection component of the manipulator; Identify the screen image and mouse position in the collected information; Locating the cursor position of the mouse and the target point position in the screen image, wherein the target point position is determined according to the operation requirements of the user; A mouse operation instruction is generated according to the mouse position, the cursor position and the target point position, and the manipulator is controlled to perform mouse automation operation based on the mouse operation instruction.

9. The method for mouse automation operation according to claim 8, characterized in that: The generating of the mouse operation instruction according to the mouse position, the cursor position and the target point position comprises: Establish a mapping relationship between the position of the mouse on the desktop and the position of the cursor on the screen; The difference between the positions of the cursor and the target point on the screen is used as input, the adjustment amount of the position of the end of the robot arm joint on the desktop is used as output, the mouse position, the cursor position and the target point position are input into the PID algorithm, and the target moving distance of the end of the robot arm joint, the target moving distance of the cursor and the click operation timing are generated according to the PID algorithm and the mapping relationship; The mouse operation instruction is generated according to the target moving distance of the end of the mechanical arm joint, the target moving distance of the cursor and the click operation timing.

10. The method for mouse automation operation according to claim 8, characterized in that: Before determining the position of the target point according to the user's operation requirements, the method further includes: Recognize the user's voice control signal or text control signal; The user's operation requirement is obtained based on the voice control signal or the text control signal.

Images