Manipulator, system and method for automatic mouse operation

Abstract

The present application relates to the field of automation technology, and in particular to a manipulator, system, and method for automated mouse operation, wherein the manipulator comprises: a manipulator arm and a collection component disposed on the manipulator arm, wherein the collection component is used to collect information and send the collected information to a host computer; a manipulator palm connected to the manipulator arm and a plurality of manipulator fingers connected to the palm; a driving component and an electric control component disposed in the manipulator palm, wherein the driving component drives any of the plurality of manipulator fingers to perform a target mouse operation, and the electric control component controls the driving component to perform automated mouse operation according to a mouse operation instruction from the host computer, wherein the host computer generates a mouse operation instruction based on the collected information. Thus, the present application solves the problem in the related art that the interaction between the manipulator and the mouse depends on the software level, the software dependence is too high, resulting in high installation complexity, poor compatibility, and poor convenience.

Description

Technical Field

[0001] The present application relates to the field of automation technology, and in particular to a manipulator, system and method for automated mouse operation. Background Art

[0002] With the rapid development of artificial intelligence, robotics, and human-computer interaction, the application of robotic arms has expanded beyond traditional industrial manufacturing to encompass more complex and detailed tasks in daily life, such as office automation and remote operation. Robotic arm control technology has also evolved from traditional physical buttons or remote controls to more sophisticated voice control and visual recognition systems. These technologies make user interaction with robotic arms more natural and intuitive, significantly improving the applicability and flexibility of robotic arms.

[0003] Currently, the interaction technology between the robot and the computer mouse mainly relies on the software level implementation, which means that the user must pre-install specific software or drivers on the controlled computer to realize the control function of the robot. This dependence will lead to the following problems:

[0004] 1. Software dependency: (1) Installation and configuration complexity: Users need to have certain technical knowledge to install and configure the necessary software. This not only increases the difficulty of use, but may also cause errors during the installation process, affecting the function of the robot. (2) Compatibility issues: The software may require specific adjustments for different operating systems or computer configurations. This customized solution limits the versatility and flexibility of the robot. 2. Lack of plug-and-play functionality: Robot control systems often do not have true plug-and-play functionality, that is, users cannot simply connect the robot to any computer and start using it immediately, but need to perform a series of settings and configurations, which is less flexible. Summary of the Invention

[0005] The present application provides a manipulator, system and method for automated mouse operation to solve the problems in related technologies in which the interaction between the manipulator and the mouse depends on the software level, the software dependence is too high, resulting in high installation complexity, poor compatibility, versatility and flexibility.

[0006] The first aspect of the present application provides a robotic arm for automated mouse operation, comprising the following steps: a robotic arm and a collection component arranged on the robotic arm, wherein the collection component is used to collect information and send the collected information to a host computer; a robotic palm connected to the robotic arm and multiple robotic fingers connected to the palm; a driving component and an electronic control component arranged in the robotic palm, wherein the driving component drives any robotic finger among the multiple robotic fingers to perform a target mouse operation, and the electronic control component controls the driving component to perform automated mouse operation according to the mouse operation instruction of the host computer, wherein the host computer generates the mouse operation instruction based on the collected information.

[0007] Optionally, the acquisition component includes at least one of a visual sensor and a sound sensor.

[0008] Optionally, the driving component includes at least one of a linear motor and a rotary motor.

[0009] Optionally, the electronic control component includes a communication module and a processor, wherein the communication module communicates with the host computer, and the processor is used to receive mouse operation instructions sent by the host computer through the communication module, and control the drive component to perform mouse automation operations according to the mouse operation instructions of the host computer.

[0010] Optionally, the electronic control component further includes a power supply for supplying power to the manipulator.

[0011] Optionally, the robotic finger is a rigid robotic finger, a soft robotic finger, or a rigid-flexible coupled robotic finger.

[0012] A second aspect of the present application provides a system for automated mouse operation, comprising: a manipulator for automated mouse operation according to the above embodiment; and a host computer, wherein the host computer communicates with the manipulator and controls the manipulator to perform automated mouse operation.

[0013] A third aspect of the present application provides a method for automated mouse operation, wherein the method performs automated mouse operation based on the manipulator for automated mouse operation of the above-mentioned embodiment, wherein the method includes the following steps: obtaining acquisition information of an acquisition component of the manipulator; identifying the screen image and mouse position in the acquisition information; locating the cursor position and target point position of the mouse in the screen image, wherein the target point position is determined according to the user's operation requirements; generating a mouse operation instruction according to the mouse position, cursor position and target point position, and controlling the manipulator to perform automated mouse operation based on the mouse operation instruction.

[0014] Optionally, a mouse operation instruction is generated based on the mouse position, cursor position and target point position, including: establishing a mapping relationship between the mouse position on the desktop and the cursor position on the screen; taking the difference between the positions of the cursor and the target point on the screen as input, and taking the adjustment amount of the position of the end of the robotic arm joint on the desktop as output, inputting the mouse position, cursor position and target point position into a PID algorithm, and generating a target moving distance of the end of the robotic arm joint, a target moving distance of the cursor and a click operation timing according to the PID algorithm and the mapping relationship; and generating a mouse operation instruction based on the target moving distance of the end of the robotic arm joint, a target moving distance of the cursor and a click operation timing.

[0015] Optionally, before determining the target point position according to the user's operation requirements, the method further includes: recognizing the user's voice control signal or text control signal; and obtaining the user's operation requirements based on the voice control signal or text control signal.

[0016] Therefore, this application has at least the following beneficial effects:

[0017] The embodiment of the present application constructs a manipulator for automatic mouse operation, in which a drive component and an electronic control component are integrated inside the manipulator. Based on the drive component and the electronic control component, the manipulator realizes automatic operation of the mouse, without the need to pre-install specific software or drivers on the controlled computer, and can achieve plug-and-play, thereby improving the convenience of using the manipulator. The electronic control component can directly control the drive component to perform automatic mouse operation according to the mouse operation instruction of the host computer, and the drive component directly controls the manipulator finger to perform the target mouse operation, reducing the dependence on software, further avoiding the compatibility problem of software installation with the computer operating system, thereby improving the versatility and flexibility of the manipulator. Thus, it solves the technical problems that the interaction between the manipulator and the mouse depends on the software level, the software dependence is too high, resulting in high installation complexity, poor compatibility, versatility and flexibility.

[0018] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0020] Figure 1 A schematic structural diagram of a manipulator for automated mouse operation according to an embodiment of the present application;

[0021] Figure 2 A hardware structure diagram of a manipulator for automated mouse operation according to an embodiment of the present application;

[0022] Figure 3 Schematic diagram of a system for automatic mouse operation according to an embodiment of the present application;

[0023] Figure 4 A specific control diagram of a system for automatic mouse operation according to an embodiment of the present application;

[0024] Figure 5 A flowchart of a method for automatic mouse operation according to an embodiment of the present application;

[0025] Figure 6 A flowchart of mouse automated game operations provided according to an embodiment of the present application;

[0026] Figure 7A schematic diagram of a human-controlled robotic arm based on a voice signal according to an embodiment of the present application;

[0027] Figure 8 This is a schematic diagram of the process of human-controlled robotic arm based on voice signals according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0029] The following describes the manipulator, system and method for automatic mouse operation according to the embodiment of the present application with reference to the accompanying drawings. In view of the problem mentioned in the above background technology that the current manipulator and computer mouse interaction technology mainly relies on the implementation at the software level, the software dependency is too high, resulting in high installation complexity, poor compatibility and convenience, the present application provides a manipulator for automatic mouse operation, in which a drive component and an electronic control component are integrated inside the manipulator. Based on the drive component and the electronic control component, the manipulator realizes the automatic operation of the mouse, without the need to pre-install specific software or drivers on the controlled computer, and can achieve plug-and-play, thereby improving the convenience of using the manipulator. The electronic control component can directly control the drive component to perform the automatic mouse operation according to the mouse operation instruction of the host computer, and the drive component directly controls the manipulator finger to perform the target mouse operation, thereby reducing the dependence on the software. Thus, the problem that the interaction between the manipulator and the mouse in the technology depends on the software level, the software dependency is too high, resulting in high installation complexity, poor compatibility, versatility and flexibility is solved.

[0030] Specifically, Figure 1 A schematic structural diagram of a manipulator for automated mouse operation provided in an embodiment of the present application.

[0031] like Figure 1 As shown, the manipulator 10 for automatic mouse operation includes: a manipulator arm 11, a collection component 12, a manipulator palm 13, a plurality of manipulator fingers 14, a driving component 15 and an electronic control component 16.

[0032] Among them, the collection component 12 is set on the robotic arm, the collection component is used to collect information and send the collected information to the host computer, the robotic palm 12 is connected to the robotic arm 11, and multiple robotic fingers 14 are connected to the robotic palm 12; the driving component 15 and the electronic control component 16 are both set in the robotic palm 12, the driving component 15 is used to drive any finger among the multiple robotic fingers 14 to perform the target mouse operation, and the electronic control component 16 controls the driving component 15 to perform the mouse automation operation according to the mouse operation instruction of the host computer, wherein the host computer generates the mouse operation instruction based on the collected information.

[0033] Among them, the multiple robotic fingers 11 can be five robotic fingers, the first to fifth fingers, and each finger performs different target mouse operations. For example, the first finger and the fifth finger are used to grasp the mouse, the second finger and the fourth finger are used to click the left and right buttons of the mouse respectively to complete single-shot, double-click, long press and other mouse operations, and the third finger is used to click the middle mouse button or scroll the wheel; the robotic arm can be any robotic arm that can meet the requirements of desktop mouse movement, without specific limitation, such as a SCARA robotic arm and other robotic arms with fewer degrees of freedom.

[0034] It can be understood that the embodiment of the present application constructs a manipulator 10 for automatic mouse operation, in which a drive component and an electronic control component are integrated inside the manipulator 10. The automatic operation of the mouse by the manipulator 10 is realized based on the drive component 15 and the electronic control component 16. There is no need to pre-install specific software or drivers on the controlled computer, and plug-and-play can be achieved, thereby improving the convenience of using the manipulator. The electronic control component 16 can directly control the drive component 15 to perform automatic mouse operation according to the mouse operation instructions of the host computer, and the drive component 15 directly controls the manipulator finger 14 to perform the target mouse operation, reducing the dependence on software, and further avoiding the compatibility issues of software installation with the computer operating system, thereby improving the versatility and flexibility of the manipulator.

[0035] In addition, it should be noted that the manipulator used for automated mouse operation is not limited to the manipulator in the embodiment of the present application, and various manipulators that can be used to operate a computer mouse fall within this scope.

[0036] In the embodiment of the present application, the acquisition component 12 includes at least one of a visual sensor and a sound sensor.

[0037] The visual sensor may be a camera, etc., and the sound sensor may be a microphone. The visual sensor and the sound sensor may also be integrated into one device, such as a hand-held camera with a microphone.

[0038] It can be understood that the collection component 12 provided on the robot arm 11 in the embodiment of the present application can be used to collect information on the screen of the computer connected to the robot arm, and the sound sensor can collect control instructions input by the user's voice, etc.

[0039] It should be noted that the visual information and auditory information collected by the collection component of the embodiment of the present application as input to the host computer is not limited to visual sensors and sound sensors, but AR and VR information can also be used as input to the host computer.

[0040] In the embodiment of the present application, the driving component 15 includes at least one of a linear motor and a rotary motor.

[0041] Among them, the number of linear motors and rotary motors can be set according to specific circumstances, for example, there are 3 linear motors and 1 rotary motor. The linear motor and rotary motor can control the linear motion and rotational motion of the robotic finger respectively.

[0042] It can be understood that the driving component 15 of the embodiment of the present application includes at least one of a linear motor and a rotary motor, and preferably both a linear motor and a rotary motor are provided to ensure precise control of the robotic finger.

[0043] In the embodiment of the present application, the electronic control component 16 includes: a communication module and a processor.

[0044] The communication module communicates with the host computer, and the processor is used to receive the mouse operation instruction sent by the host computer through the communication module, and control the driving component 15 to perform the mouse automation operation according to the mouse operation instruction of the host computer.

[0045] It can be understood that the electric component 16 of the embodiment of the present application includes a communication module and a processor. The communication module is used to control the manipulator to communicate with the host computer, specifically for wireless communication via Bluetooth. The processor receives the mouse operation instructions sent by the host computer through the communication module, and controls the drive component 15 to perform mouse automation operations according to the mouse operation instructions of the host computer. The processor can be a control board, such as STM32F407.

[0046] In the embodiment of the present application, the electronic control component 16 also includes a power supply for supplying power to the manipulator.

[0047] The power source may be a battery.

[0048] In the embodiment of the present application, the robotic finger 14 is a rigid robotic finger, a soft robotic finger, or a rigid-flexible coupled robotic finger.

[0049] It is understandable that the robotic finger 14 of the embodiment of the present application may be a rigid robotic finger, a soft robotic finger, or a rigid-flexible coupled robotic finger to improve the applicability of the robotic arm.

[0050] The following describes a specific embodiment of the present invention to describe the robot for automatic mouse operation. The specific hardware structure is as follows: Figure 2 shown.

[0051] The robotic hand consists of two main parts: fingers and a palm. It has five fingers in total. Fingers 1 and 5 are used to grasp the mouse and can adapt to different mouse sizes. Fingers 2 and 4 are used for the left and right buttons of the motorized mouse, which can perform mouse operations such as single-click, double-click, and long press. Fingers 3 is used for the middle button of the motorized mouse or scrolling the scroll wheel.

[0052] The palm contains the robot's drive and electronic control components, mainly including 3 linear motors and 1 rotary motor. The Bluetooth module is used for wireless communication, the control board (equivalent to the above-mentioned processor) is used to control the movement of the robot, and the battery is used to provide energy.

[0053] In summary, the manipulator of the embodiment of the present application can be plug-and-play, so that it can be seamlessly connected to computers of various brands and models without the need for users to perform complex software installation and configuration. In addition, the manipulator control system can run on a variety of operating systems, whether it is Windows, MacOS or Linux, to ensure stability and compatibility. This greatly reduces the time and technical barriers for users in setting up the device and improves the user experience. And the plug-and-play feature makes the manipulator easier to be accepted and used by different groups of people, especially for people with disabilities and elderly users, who can more easily control the computer through voice or simple operations.

[0054] According to the manipulator for automatic mouse operation proposed in the embodiment of the present application, a manipulator for automatic mouse operation is constructed, in which a drive component and an electronic control component are integrated inside the manipulator. The manipulator realizes automatic operation of the mouse based on the drive component and the electronic control component. There is no need to pre-install specific software or drivers on the controlled computer, and plug-and-play can be achieved, thereby improving the convenience of using the manipulator. The electronic control component can directly control the drive component to perform automatic mouse operation according to the mouse operation instructions of the host computer, and the drive component directly controls the manipulator finger to perform the target mouse operation, reducing the dependence on software, and further avoiding the compatibility issues of software installation with the computer operating system, thereby improving the versatility and flexibility of the manipulator.

[0055] An embodiment of the present application also provides a system for automatic mouse operation.

[0056] like Figure 3 As shown, the system 20 for automatic mouse operation includes: the above-mentioned manipulator 10 for automatic mouse operation and a host computer 21.

[0057] The host computer 21 communicates with the manipulator 10 and controls the manipulator 10 to perform mouse automation operations.

[0058] It can be understood that the embodiment of the present application can control the manipulator to perform automated operations through the host computer to realize the control function of the manipulator, wherein the host computer can communicate with the manipulator via Bluetooth.

[0059] Specifically, the specific control diagram of the system for mouse automatic operation in the embodiment of the present application is as follows: Figure 4 The host computer receives the collected information (such as visual signals and voice signals) from the collection component and controls the manipulator via Bluetooth.

[0060] Next, a method for automatic mouse operation according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0061] Figure 5 This is a flowchart of a method for automated mouse operation according to an embodiment of the present application.

[0062] like Figure 5 As shown, the method for mouse automation operation, based on the above-mentioned manipulator for mouse automation operation, performs mouse automation operation, including the following steps:

[0063] In step S101 , acquisition information of the acquisition component of the robot is acquired.

[0064] The collected information may be obtained by the above-mentioned collecting components, including computer screen information, user voice commands, and the like.

[0065] In step S102, the screen image and the mouse position in the collected information are identified.

[0066] The screen information specifically includes screen image and mouse information.

[0067] In step S103 , the cursor position of the mouse and the target point position on the screen are located, wherein the target point position is determined according to the user's operation requirements.

[0068] The position of the target point may be determined based on the user's operational requirements, for example, the position of the target point may be a target in a game, or the position of an icon on a screen.

[0069] Among them, the specific method of positioning can be selected according to the specific situation. Taking the positioning of the target as an example, the screen image is perspective transformed to obtain a 1920×1080 frame, and the predefined target pattern is convolved with the frame. The point with the highest convolution value is the position of the corresponding icon, thereby obtaining the position of the cursor and the target.

[0070] In an embodiment of the present application, before determining the target point position according to the user's operation requirements, it also includes: identifying the user's voice control signal or text control signal; and obtaining the user's operation requirements based on the voice control signal or text control signal.

[0071] It can be understood that the embodiments of the present application can recognize the user's voice control signals or text control signals, and then obtain the user's operation requirements based on the voice control signals or text control signals, thereby improving the flexibility and user-friendliness of the operation.

[0072] Specifically, the embodiment of the present application can obtain the user's voice control instructions through a microphone, and further convert and analyze the voice control signal into a text control signal, or directly set the text information through the host computer to generate a text control signal.

[0073] In step S104 , a mouse operation instruction is generated according to the mouse position, the cursor position, and the target point position, and the manipulator is controlled to perform the mouse automation operation based on the mouse operation instruction.

[0074] It is understandable that the embodiments of the present application can generate mouse operation instructions based on the mouse position, cursor position and target point position, and control the manipulator to perform mouse automation operations based on the mouse operation instructions.

[0075] In an embodiment of the present application, a mouse operation instruction is generated based on the mouse position, the cursor position and the target point position, including: establishing a mapping relationship between the mouse position on the desktop and the cursor position on the screen; taking the difference between the positions of the cursor and the target point on the screen as input, and taking the adjustment amount of the position of the end of the robotic arm joint on the desktop as output, inputting the mouse position, the cursor position and the target point position into a PID algorithm, and generating the target moving distance of the end of the robotic arm joint, the target moving distance of the cursor and the click operation timing according to the PID algorithm and the mapping relationship; generating a mouse operation instruction based on the target moving distance of the end of the robotic arm joint, the target moving distance of the cursor and the click operation timing.

[0076] It can be understood that the embodiment of the present application can establish a mapping relationship between the position of the mouse on the desktop and the position of the cursor on the screen to reduce the number of planning times of the robotic arm and ensure that the cursor can move to the target position quickly and accurately. The difference between the position of the cursor and the target point on the screen frame is used as input, and the adjustment amount of the position of the end of the robotic arm joint on the desktop is used as output. The mouse position, cursor position and the target point position are input into the PID algorithm, and the target movement distance of the end of the robotic arm joint, the target movement distance of the cursor and the click operation timing are generated according to the PID algorithm and the mapping relationship. Then, the mouse operation instruction is generated according to the target movement distance of the end of the robotic arm joint, the target movement distance of the cursor and the click operation timing to realize the operation of the target point.

[0077] Specifically, the PID algorithm process is as follows: the difference between the positions of the cursor and the target point on the frame is used as input, and the adjustment amount of the position of the end of the robotic arm joint on the desktop is used as output. When the cursor does not move to the target point, the end of the robotic arm joint is controlled to move a specific distance; when it is detected that the two overlap, a click operation is performed.

[0078] Among them, the mapping relationship is constructed to establish a mapping relationship between the position of the cursor on the screen and the position of the mouse on the desktop, to control the movement speed of the end of the robotic arm joint. The distance the cursor position moves on the screen is proportional to the distance the mouse moves on the desktop. The ratio of the two can be measured experimentally, and thus the mapping relationship between the two is established: y = ax, and a PID algorithm is established based on the mapping relationship. The deviation between the observed cursor position and the target position on the screen frame is used as input, and the proportional coefficient of the mapping relationship is used as the proportional term in the PID algorithm. The calculated output is the distance the mouse needs to move (the distance the end of the robotic arm joint needs to move).

[0079] It should be noted that the aforementioned explanations of the embodiment of the manipulator and system for automatic mouse operation are also applicable to the method for automatic mouse operation of this embodiment, and will not be repeated here.

[0080] The method for automatic mouse operation is described below through specific embodiments.

[0081] Example 1: Automated operation based on visual signals. Take the mouse automated game operation as an example. The target point is the target in the game. The specific process is as follows: Figure 6 As shown, including:

[0082] 1. Initialization.

[0083] The system, which deploys cameras, manipulators, and robotic arms, is initially in an arbitrary random position. When the entire system receives instructions from the control host computer, it is initialized. The initialization mainly controls the robotic arm to move to a specific predefined posture.

[0084] 2. Find the position of the mouse on the two-dimensional plane.

[0085] Start the camera and find the position of the mouse on the two-dimensional plane, that is, the position of the mouse on the desktop.

[0086] 3. After image processing, the robotic arm is controlled to grab the mouse.

[0087] After image processing, the robot arm is controlled to grab the mouse and adjusted to the optimal grasping position.

[0088] 4. Adjust the position of the robot arm so that the camera on the back of the hand is aimed at the computer screen connected to the mouse.

[0089] 5. Start the target shooting game.

[0090] 6. Extract the edge of the computer screen.

[0091] A visual algorithm was developed based on OpenCV. First, the edge of the image information captured by the camera was extracted to determine whether there is a computer screen in the image and determine the location of the computer screen.

[0092] 7. Perspective transformation to obtain a two-dimensional screen.

[0093] If the computer screen is in the frame, the screen image is perspective transformed to obtain a 1920×1080 frame.

[0094] 8. Convolution positioning cursor and target.

[0095] The predefined pattern is convolved with the frame, and the point with the highest convolution value is the position of the corresponding icon, thereby obtaining the position of the cursor and target.

[0096] 9. Control the robotic arm to move the cursor close to the target.

[0097] 10. Determine whether the cursor coincides with the target.

[0098] 11. If they overlap, control the robot arm to complete the click operation; otherwise, control the robot arm again to move the cursor closer to the target.

[0099] 12. Repeat the above process until the host computer sends a control command to end the game.

[0100] Example 2: Human control of a robot for automatic mouse operation based on voice signals, such as Figure 7 shown.

[0101] In this embodiment of the application, a real-time speech recognition function is introduced, and the system can detect the instructions issued by the operator and convert them into text information. Basic functional functions such as controlling the movement of the end of the robot arm and controlling the operation of the robot hand are pre-defined, and relevant text is written to introduce the task background and function functions. These application programming interfaces (APIs), background descriptions, and commands given by the operator will constitute the prompts passed to ChatGPT. The large language model ChatGPT will then sort out and disassemble the logical relationship of the task.

[0102] Example 3: The execution process of controlling the robot to open the browser, close the browser and shut down the computer based on the voice signal, such as Figure 8 shown.

[0103] A voice control task was used as an example to demonstrate ChatGPT's ability to understand user commands and generate corresponding code. The operator gave the following instructions: open a browser on the desktop, close the browser, and turn off the computer.

[0104] 1. Open the browser: ChatGPT first receives the command "Open the browser on the desktop" and identifies the key action word "open" and the object "browser" through natural language processing technology.

[0105] Task decomposition: ChatGPT decomposes the task into the following two steps:

[0106] 1. Move the cursor: Use visual servo control technology to identify the location of the browser icon on the desktop, and use the control algorithm to calculate the cursor path to move to the icon.

[0107] 2. Open the application: Use the command of double-clicking the left mouse button to activate and open the browser.

[0108] Function call and execution: Call predefined APIs such as move_to("Internet") and double_click_left_button() to complete the operation. At the same time, the system monitors the operation status in real time to ensure accurate execution of the task.

[0109] 2. Close the browser: Input "close the browser" through voice, and ChatGPT will parse "close" and "browser".

[0110] Task decomposition: broken down into the following steps:

[0111] 1. Locate the close button: Move the cursor to the upper right corner of the screen.

[0112] 2. Execute the close operation: Move the cursor to the close button and click the left mouse button.

[0113] 3. Turn off the computer.

[0114] Task breakdown: This instruction is broken down into three steps:

[0115] 1. Launch the Start menu: Move your mouse to the Start button in the lower left corner of the screen.

[0116] 2. Navigate to Power Options: Locate the power button in the Start menu.

[0117] 3. Perform the shutdown operation: Select "Shut down" in the power options and confirm the operation.

[0118] According to the method for mouse automation operation proposed in the embodiment of the present application, the screen image and mouse position in the information collected by the manipulator can be identified, and the mouse position and target point position in the screen image can be located. The user can determine the position of the target point according to simple operation requirements, generate mouse operation instructions according to the mouse position, cursor position and target point position, and then control the manipulator to perform mouse automation operation based on the mouse operation instructions. The degree of automation is high and the user does not need to perform complicated operations.

[0119] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0120] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0121] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0122] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0123] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

Claims

1. A manipulator for automatic mouse operation, characterized in that: include: A robotic arm and a collection component provided on the robotic arm, wherein the collection component is used to collect information and send the collected information to a host computer; a robotic palm connected to the robotic arm and a plurality of robotic fingers connected to the palm; A driving component and an electronic control component are arranged in the palm of the robotic hand, wherein the driving component drives any robotic finger among the multiple robotic fingers to perform a target mouse operation, and the electronic control component controls the driving component to perform an automated mouse operation according to a mouse operation instruction of the host computer, wherein the host computer generates the mouse operation instruction according to the collected information.

2. The manipulator for automatic mouse operation according to claim 1, characterized in that: The collecting component includes at least one of a visual sensor and a sound sensor.

3. The manipulator for automatic mouse operation according to claim 1, characterized in that: The driving component includes at least one of a linear motor and a rotary motor.

4. The manipulator for automatic mouse operation according to claim 1, characterized in that: The electronic control component includes a communication module and a processor, wherein the communication module communicates with the host computer, and the processor is used to receive mouse operation instructions sent by the host computer through the communication module, and control the drive component to perform mouse automation operations according to the mouse operation instructions of the host computer.

5. The manipulator for automatic mouse operation according to claim 4, characterized in that: The electronic control component also includes a power supply for supplying power to the manipulator.

6. The manipulator for automatic mouse operation according to claim 1, characterized in that: The mechanical finger is a rigid mechanical finger, a soft mechanical finger or a rigid-flexible coupled mechanical finger.

7. A system for automatic mouse operation, characterized in that: include: The manipulator for automatic mouse operation according to any one of claims 1 to 6; A host computer, wherein the host computer communicates with the manipulator and controls the manipulator to perform mouse automation operations.

8. A method for automatic mouse operation, characterized in that: The method performs mouse automation operation based on the manipulator for mouse automation operation according to any one of claims 1 to 6, wherein the method comprises the following steps: Obtaining collection information of the collection component of the manipulator; Identifying the screen image and mouse position in the collected information; Locating a mouse cursor position and a target point position on the screen, wherein the target point position is determined according to a user's operation requirement; A mouse operation instruction is generated according to the mouse position, the cursor position and the target point position, and the manipulator is controlled to perform mouse automation operation based on the mouse operation instruction.

9. The method for mouse automation operation according to claim 8, characterized in that: Generating a mouse operation instruction according to the mouse position, the cursor position, and the target point position includes: Establish a mapping relationship between the position of the mouse on the desktop and the position of the cursor on the screen; The difference between the positions of the cursor and the target point on the screen is used as input, and the adjustment amount of the position of the end joint of the robotic arm on the desktop is used as output. The mouse position, the cursor position, and the target point position are input into a PID algorithm, and a target movement distance of the end joint of the robotic arm, a target movement distance of the cursor, and a click operation timing are generated according to the PID algorithm and the mapping relationship. The mouse operation instruction is generated according to the target moving distance of the end of the mechanical arm joint, the target moving distance of the cursor and the click operation timing.

10. The method for mouse automation operation according to claim 8, characterized in that: Before determining the target point position according to the user's operation requirements, the method further includes: Recognize the user's voice control signal or text control signal; The user's operation requirement is obtained based on the voice control signal or the text control signal.

Images