Mechanical arm control method and system based on time sequence visual operation manual

Abstract

The invention belongs to the technical field of mechanical arm control. The invention provides a mechanical arm control method and system based on a time sequence visual operation manual, and the method comprises the steps: carrying out the preprocessing and target recognition of an obtained on-site image, and obtaining a target recognition result; according to the target recognition result, retrieval is conducted in a predefined visual operation manual, and a time sequence visual operation set of the mechanical arm is obtained; and path planning is conducted according to the time sequence visual operation set, and action control over the mechanical arm is conducted according to the path planning result. According to the method, a series of continuous images are adopted as operation instructions, the images can clearly display each step of the operation process and cover the complex scene of the operation of the power equipment, the time sequence image not only comprises the current step, but also can provide the sequence and rhythm of the operation, and the continuity of the task is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of robot arm control, and in particular to a robot arm control method and system based on a time series visual operation manual. Background Art

[0002] The statements in this section merely provide background art related to the present invention and do not necessarily constitute prior art.

[0003] Robotic arms have been widely used in the field of electric robots. Although the shapes of robotic arms are different, they all have one common feature, which is the ability to accurately locate a point in three-dimensional (or two-dimensional) space to perform operations according to preset fixed actions or programmed instructions.

[0004] The existing robotic arm control in the power field still has the following problems: (1) The operation scenarios of power equipment are relatively complex, and the temperature or electromagnetic interference in the environment will affect the control performance of the robotic arm; (2) In order to cope with environmental uncertainty and complexity, the existing robotic arm control algorithms often require a high amount of calculation and have a high computational cost. Summary of the invention

[0005] In order to address the deficiencies of the prior art, the present invention provides a robotic arm control method and system based on a time series visual operation manual, which solves the problems of complex calculations and high computational costs of existing robotic arm control algorithms, avoids complex programming instructions during robotic arm operation, and ensures the continuity of tasks.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: In a first aspect, the present invention provides a robotic arm control method based on a time series visual operation manual.

[0007] A robotic arm control method based on a time series visual operation manual includes the following processes: Preprocessing and target recognition are performed on the acquired on-site images to obtain target recognition results; According to the target recognition result, searching in a predefined visual operation manual to obtain a time series visual operation set of the robot arm; Path planning is performed according to the time series visual operation set, and motion control of the robotic arm is performed according to the path planning result.

[0008] As a further limitation of the first aspect of the present invention, the target recognition result includes: key components, tools and relative positions between the key components and tools in the scene image.

[0009] As a further limitation of the first aspect of the present invention, the construction of the predefined visual operation manual includes: Acquire images captured by the camera at each moment when the robotic arm is manually operated to complete a specified action; Sort the image sequence according to the time axis to obtain an image sequence; The image sequence is mapped to a corresponding operation instruction set to obtain a predefined visual operation manual.

[0010] As a further limitation of the first aspect of the present invention, hash mapping is used to search in a predefined visual operation manual. When the retrieval match is successful, the time series visual operation set of the robot arm is extracted, and the time series visual operation set includes: the three-dimensional coordinates of the power equipment components that need to be operated, the matching tool name and usage method, and the position and order of key components in the entire operation process.

[0011] As a further limitation of the first aspect of the present invention, based on the motion planning algorithm, the optimal path from the current position of the robot arm to the target position is calculated, and based on the optimal path, a detailed motion trajectory is generated, and inverse kinematics is used to calculate the angles of each joint of the robot arm. According to the task requirements, the speed and acceleration parameters are set to make the movement of the robot arm smooth and safe.

[0012] As a further limitation of the first aspect of the present invention, an image of the robot arm during operation is captured in real time, and the image is compared with an expected action image in a time series visual operation set of the robot arm, and the error between the actual operation and the expected action is calculated: PID control is used to perform adaptive adjustments based on the error value and update the operation instructions in real time.

[0013] In a second aspect, the present invention provides a robotic arm control system based on a time series visual operation manual.

[0014] A robotic arm control system based on a time series visual operation manual, comprising: The target recognition unit is configured to: pre-process and recognize the target on the acquired scene image to obtain a target recognition result; An operation set retrieval unit is configured to: search in a predefined visual operation manual according to the target recognition result to obtain a time series visual operation set of the robot arm; The motion control unit is configured to: perform path planning according to the time series visual operation set, and perform motion control of the robotic arm according to the path planning result.

[0015] In a third aspect, the present invention provides a computer device, comprising: a processor and a computer-readable storage medium; a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the robot arm control method based on the time series visual operation manual as described in the first aspect of the present invention is implemented.

[0016] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by a processor and executing the robotic arm control method based on a time series visual operation manual as described in the first aspect of the present invention.

[0017] In a fifth aspect, the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the robot arm control method based on the time series visual operation manual as described in the first aspect of the present invention.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention innovatively proposes a robotic arm control method based on a time series visual operation manual, which solves the problems of complex calculations and high computational costs of existing robotic arm control algorithms, and avoids complex programming instructions when operating the robotic arm; each image is analyzed through visual recognition technology to extract key operation information, such as the position of the target component, tool requirements for operation, and motion trajectory, etc. In this way, the robotic arm can adaptively adjust the operation strategy according to the dynamic changes in the image, and respond to task changes in a complex environment in real time; a series of continuous time series images are used as operation instructions, which not only include the current step, but also provide the order and rhythm of the operation, and can clearly display each step of the operation process, covering complex scenarios of power equipment operation, such as equipment inspection, changes in operation steps, etc., so that the robotic arm can seamlessly connect multiple operation steps, ensure the continuity of the task, and realize automated and continuous task execution, which is suitable for various complex power operation scenarios, such as multi-step equipment maintenance or installation processes.

[0019] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0021] Figure 1 A schematic flow chart of a method for controlling a robotic arm based on a time series visual operation manual provided in Example 1 of the present invention; Figure 2A schematic diagram of a robotic arm control system based on a time series visual operation manual provided in Example 2 of the present invention; Figure 3 A schematic diagram of a computer device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0022] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0023] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0024] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0025] Embodiment 1: This implementation proposes a robotic arm control method based on a time series visual operation manual, such as Figure 1 As shown, the following process is included: S1: preprocessing and target recognition of the acquired on-site image to obtain the target recognition result; S2: According to the target recognition result, a search is performed in a predefined visual operation manual to obtain a time series visual operation set of the robot arm; S3: Perform path planning according to the time series visual operation set, and perform motion control of the robotic arm according to the path planning result.

[0026] In this implementation, the creation of a predefined visual operation manual includes: First, manually operate the robot arm to complete the specified action, and use the camera to obtain image data, which is represented as a pixel value matrix ,in t Is a timestamp, representing the moment in the image sequence t Images: (1); in, The dimension is H × W , H and W are the height and width of the image, Indicated in Middle i Row, No. j The pixel value at the column; Second, sort the image sequence according to the time axis: (2); Among them, S represents the entire time series image set, T is the total number of time steps, and these images represent different time nodes of the robot operation. Each image Corresponding to a specific step of the operation, the timeline provides the continuity and sequence of the operation; Finally, create a timeline guide for the operation.

[0027] To ensure that the robot can perform operations in sequence, we define a mapping function , the time t Mapped to the corresponding operation instruction set : (3); in, is a mapping function from images to operation instructions (which can be based on the results of visual recognition algorithms), and the operation instruction set Including information about position, angle and task execution, as well as the operating instruction set at each moment A manual for visual manipulation of time series.

[0028] In this implementation, the robot arm's operation is completed according to the predefined visual operation manual, specifically including: Image acquisition and key component identification: The camera acquires image information, analyzes the image based on the SAM image recognition algorithm, and identifies the key components, tools and their relative positions in the power equipment. Image acquisition: The camera captures images of the equipment site at a certain frequency and generates an image data matrix; Image preprocessing: Perform preprocessing operations such as filtering and denoising on the image to ensure that the image quality meets the recognition requirements; Target segmentation and recognition: The SAM model segments the targets in the image, divides the image into multiple regions, and analyzes each region to identify key components (such as transformers, switches, insulators, etc.) and tools (such as wrenches, screwdrivers, etc.) in power equipment, as well as their relative positions.

[0029] This information is represented in matrix form: (4); in is the coordinate position of key parts and tools in the image, and n is the number of detected targets.

[0030] Retrieval of visual operation manual and extraction of operation information. The system searches the visual operation manual based on the image information to obtain the operation information. After completing the target recognition, the system will match the recognized information with the predefined visual operation manual to obtain the corresponding operation steps. The specific steps are as follows: Retrieval algorithm: Based on the target position and category, the system searches for the corresponding operation information in the predefined database or manual. The retrieval system uses hash mapping to quickly locate the required operation steps. Operation information extraction, once the match is successful, the system extracts the operation information set ,include: Target location: the three-dimensional coordinates of the power equipment component that needs to be operated; Tool selection: matching tool names and usage methods; Operation sequence: the position and order of the component in the entire operation process.

[0031] Path planning and motion control: Based on the operation information analyzed from the image, the robot arm plans the path through the motion planning algorithm to set the precise motion trajectory and range of motion. Path planning algorithm: The system uses the A* motion planning algorithm to calculate the optimal path from the current position of the robot arm to the target position, ensuring that obstacles are avoided and the motion time is optimized.

[0032] Trajectory generation: Generate a detailed motion trajectory based on the planned path, use inverse kinematics to calculate the angles of each joint of the robot arm, and generate : (5); in is the three-dimensional coordinate of the target position, is the angle set of each joint.

[0033] Motion range control: According to the task requirements, set appropriate speed and acceleration parameters to ensure smooth and safe movement of the robot arm.

[0034] Real-time image feedback and adaptive adjustment. The action planning process is combined with real-time image feedback to ensure that each operation of the robot arm can accurately aim at the target and adjust according to slight changes in the environment to avoid misoperation. During operation, the system will monitor the movement of the robot arm in combination with real-time image feedback and make necessary adjustments according to environmental changes to ensure operation accuracy. Real-time image acquisition: The camera continuously captures images of the robot arm operation and compares them with the expected actions in the operation manual; Error detection: by calculating the error between the actual operation and the expected action: (6); in, and are the actual position and the expected position respectively.

[0035] Action adjustment: according to the error value use PID The control is adaptively adjusted and the operation instructions are updated in real time to ensure that the robot arm can accurately aim at the target every time it operates: (7); in, and To control the gain, adjust the response speed and accuracy of the system.

[0036] Through the above process, the robotic arm can automatically perform precise operating tasks in complex power scenarios and make adaptive adjustments based on real-time changes to ensure safe and efficient operations.

[0037] Embodiment 2: like Figure 2 As shown, this implementation provides a robotic arm control system based on a time series visual operation manual, including: The target recognition unit is configured to: pre-process and recognize the target on the acquired scene image to obtain a target recognition result; An operation set retrieval unit is configured to: search a predefined visual operation manual according to the target recognition result to obtain a time series visual operation set of the robot arm; The motion control unit is configured to: perform path planning according to the time series visual operation set, and perform motion control of the robotic arm according to the path planning result.

[0038] The specific working process of each of the above units is described in Example 1 and will not be repeated here.

[0039] It is understandable that the above-mentioned units can be separately or completely combined into one or several other units to constitute, or one (some) of the units can be further divided into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the system may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0040] According to another embodiment of the present application, the system described in this embodiment can be constructed, and the method of Example 1 of the present application can be implemented by running a computer program (including program code) capable of executing the steps involved in the corresponding method described in Example 1 on a general-purpose computing device such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0041] Embodiment 3: like Figure 3 As shown, this implementation provides an electronic device, which includes a processor 1001, a communication interface 1002, and a computer-readable storage medium 1003. The processor 1001, the communication interface 1002, and the computer-readable storage medium 1003 may be connected via a bus or other means.

[0042] Among them, the communication interface 1002 is used to receive and send data, the computer-readable storage medium 1003 can be stored in the memory of the electronic device, the computer-readable storage medium 1003 is used to store a computer program, the computer program includes program instructions, and the processor 1001 is used to execute the program instructions stored in the computer-readable storage medium 1003.

[0043] The processor 1001 (or CPU (Central Processing Unit)) is the computing core and control core of the electronic device, which is suitable for implementing one or more instructions, and specifically suitable for loading and executing one or more instructions to implement corresponding method processes or corresponding functions.

[0044] The processor 1001 is configured to execute the following process: Preprocessing and target recognition are performed on the acquired on-site images to obtain target recognition results; According to the target recognition result, searching in a predefined visual operation manual to obtain a time series visual operation set of the robot arm; Path planning is performed according to the time series visual operation set, and motion control of the robotic arm is performed according to the path planning result.

[0045] The specific working process is described in Example 1 and will not be repeated here.

[0046] Embodiment 4: This implementation provides a computer-readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the electronic device and, of course, extended storage media supported by the electronic device. The computer-readable storage medium provides a storage space that stores the processing system of the electronic device.

[0047] In addition, the storage space also stores one or more instructions suitable for being loaded and executed by the processor, and these instructions may be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here may be a high-speed RAM memory, or a non-volatile memory, such as at least one disk memory; optionally, it may also be at least one computer-readable storage medium located away from the aforementioned processor.

[0048] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to implement the following process: Preprocessing and target recognition are performed on the acquired on-site images to obtain target recognition results; According to the target recognition result, searching in a predefined visual operation manual to obtain a time series visual operation set of the robot arm; Path planning is performed according to the time series visual operation set, and motion control of the robotic arm is performed according to the path planning result.

[0049] The specific working process is described in Example 1 and will not be repeated here.

[0050] Embodiment 5: The present implementation provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the following process: Preprocessing and target recognition are performed on the acquired on-site images to obtain target recognition results; According to the target recognition result, searching in a predefined visual operation manual to obtain a time series visual operation set of the robot arm; Path planning is performed according to the time series visual operation set, and motion control of the robotic arm is performed according to the path planning result.

[0051] The specific working process is described in Example 1 and will not be repeated here.

[0052] A person skilled in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0053] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by the computer or a data processing device such as a server, a data center, etc. that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0054] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A robotic arm control method based on a time series visual operation manual, characterized in that: The process includes: Preprocessing and target recognition are performed on the acquired on-site images to obtain target recognition results; According to the target recognition result, searching in a predefined visual operation manual to obtain a time series visual operation set of the robot arm; Path planning is performed according to the time series visual operation set, and motion control of the robotic arm is performed according to the path planning result.

2. The robotic arm control method based on the time series visual operation manual according to claim 1, characterized in that: The target recognition result includes: key components, tools and relative positions between the key components and tools in the scene image.

3. The robotic arm control method based on time series visual operation manual according to claim 1, characterized in that: Construction of predefined vision playbooks, including: Acquire images captured by the camera at each moment when the robotic arm is manually operated to complete a specified action; Sort the image sequence according to the time axis to obtain an image sequence; The image sequence is mapped to a corresponding operation instruction set to obtain a predefined visual operation manual.

4. The robotic arm control method based on time series visual operation manual according to claim 1, characterized in that: Hash mapping is used to search in the predefined visual operation manual. When the retrieval matches successfully, the time series visual operation set of the robot arm is extracted. The time series visual operation set includes: the three-dimensional coordinates of the power equipment components that need to be operated, the matching tool name and usage method, and the position and order of key components in the entire operation process.

5. The robotic arm control method based on time series visual operation manual according to claim 1, characterized in that: Based on the motion planning algorithm, the optimal path from the current position of the robot to the target position is calculated. Based on the optimal path, a detailed motion trajectory is generated, and inverse kinematics is used to calculate the angles of each joint of the robot. According to the task requirements, the speed and acceleration parameters are set to make the movement of the robot smooth and safe.

6. The robotic arm control method based on time series visual operation manual according to claim 1, characterized in that: Capture the image of the robot arm during operation in real time, compare it with the expected action image in the robot arm's time series visual operation set, and calculate the error between the actual operation and the expected action: PID control is used to perform adaptive adjustments based on the error value and update the operation instructions in real time.

7. A robotic arm control system based on a time series visual operation manual, characterized in that: include: The target recognition unit is configured to: pre-process and recognize the target on the acquired scene image to obtain a target recognition result; An operation set retrieval unit is configured to: search a predefined visual operation manual according to the target recognition result to obtain a time series visual operation set of the robot arm; The motion control unit is configured to: perform path planning according to the time series visual operation set, and perform motion control of the robotic arm according to the path planning result.

8. A computer device, characterized in that: include: a processor and a computer readable storage medium; a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the robot arm control method based on the time series visual operation manual as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the robot arm control method based on a time series visual operation manual as described in any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the robot arm control method based on the time series visual operation manual as described in any one of claims 1 to 6.

Images