Robotic arm control method and system based on time series visual operation manual

Abstract

The application belongs to the technical field of mechanical arm control. A mechanical arm control method and system based on time sequence visual operation manual are provided. The acquired live images are preprocessed and target recognition is performed to obtain a target recognition result. According to the target recognition result, a search is performed in a predefined visual operation manual to obtain a time sequence visual operation set of the mechanical arm. Path planning is performed according to the time sequence visual operation set, and motion control of the mechanical arm is performed according to the path planning result. The application uses a series of continuous images as operation instructions, which can clearly show each step of the work flow and cover the complex scene of power equipment work. The time sequence images not only contain the current step, but also provide the sequence and rhythm of the operation, ensuring the continuity of the task.

Description

Technical Field

[0001] This invention relates to the field of robotic arm control technology, specifically to a robotic arm control method and system based on a time-series vision operation manual. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Robotic arms have been widely used in the field of electric robots. Although robotic arms come in different shapes, they all have one thing in common: they can accurately position themselves at a point in three-dimensional (or two-dimensional) space to perform tasks according to preset fixed actions or programming instructions.

[0004] The existing robotic arm control in the power field still has the following problems: (1) The operation scenarios of power equipment are relatively complex, and the temperature or electromagnetic interference in the environment will affect the control performance of the robotic arm; (2) In order to cope with the uncertainty and complexity of the environment, the existing robotic arm control algorithms often require a high amount of computation, resulting in high computational costs. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a robotic arm control method and system based on a time-series visual operation manual. This method solves the problems of computational complexity and high computational cost in existing robotic arm control algorithms, avoids complex programming instructions during robotic arm operation, and ensures the continuity of tasks.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a robotic arm control method based on a time-series visual operation manual.

[0007] A robotic arm control method based on a time-series vision operation manual includes the following processes: The acquired scene images are preprocessed and target recognition is performed to obtain the target recognition results; Based on the target recognition results, a time-series visual operation set for the robotic arm is obtained by searching a predefined visual operation manual. Path planning is performed based on the time-series visual operation set, and the motion control of the robotic arm is performed based on the path planning results.

[0008] As a further limitation of the first aspect of the present invention, the target recognition result includes: key components and tools in the scene image, and the relative positions between the key components and the tools.

[0009] As a further limitation of the first aspect of the invention, the construction of the predefined visual operation manual includes: Acquire images captured by the camera at various moments when the robotic arm is manually operated to complete a specified action; The image sequence is obtained by sorting the image sequence according to the time axis; The image sequence is mapped to the corresponding operation instruction set to obtain a predefined visual operation manual.

[0010] As a further limitation of the first aspect of the present invention, a hash mapping is used to search in a predefined visual operation manual. When the search is successful, the time-series visual operation set of the robotic arm is extracted. The time-series visual operation set includes: the three-dimensional coordinates of the power equipment components to be operated, the name and usage method of the matching tool, and the position and order of the key components in the entire operation process.

[0011] As a further limitation of the first aspect of the present invention, based on the motion planning algorithm, the optimal path from the current position of the robotic arm to the target position is calculated. Based on the optimal path, a detailed motion trajectory is generated, and the angles of each joint of the robotic arm are calculated using inverse kinematics. According to the task requirements, the speed and acceleration parameters are set to make the movement of the robotic arm smooth and safe.

[0012] As a further limitation of the first aspect of the invention, images of the robotic arm operation are captured in real time and compared with the expected action images in the time-series visual operation set of the robotic arm to calculate the error between the actual operation and the expected action: The system uses PID control to adaptively adjust based on the error value and updates the operating instructions in real time.

[0013] Secondly, the present invention provides a robotic arm control system based on a time-series vision operation manual.

[0014] A robotic arm control system based on a time-series vision operation manual includes: The target recognition unit is configured to: preprocess the acquired scene image and perform target recognition to obtain the target recognition result; The operation set retrieval unit is configured to: search in a predefined visual operation manual based on the target recognition result to obtain the time-series visual operation set of the robotic arm; The motion control unit is configured to: perform path planning based on the time-series visual operation set, and perform motion control of the robotic arm based on the path planning result.

[0015] Thirdly, the present invention provides a computer device, comprising: a processor and a computer-readable storage medium; A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the robotic arm control method based on a time-series visual operation manual as described in the first aspect of the present invention.

[0016] Fourthly, the present invention provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed as described in the first aspect of the present invention: a robotic arm control method based on a time-series vision operation manual.

[0017] Fifthly, the present invention provides a computer program product comprising a computer program that, when executed by a processor, implements the robotic arm control method based on a time-series visual operation manual as described in the first aspect of the present invention.

[0018] Compared with the prior art, the beneficial effects of the present invention are: This invention innovatively proposes a robotic arm control method based on a time-series visual operation manual, solving the problems of computational complexity and high cost of existing robotic arm control algorithms, and avoiding complex programming instructions during robotic arm operation. By analyzing each image using visual recognition technology, key operational information such as the position of the target component, tool requirements, and motion trajectory are extracted. In this way, the robotic arm can adaptively adjust its operation strategy according to dynamic changes in the image, responding in real time to task changes in complex environments. Using a series of continuous time-series images as operation instructions not only includes the current step but also provides the order and rhythm of the operation, clearly displaying each step of the work process. This covers complex scenarios in power equipment operation, such as equipment inspection and changes in operation steps, enabling the robotic arm to seamlessly connect multiple operation steps, ensuring task continuity, and achieving automated and continuous task execution. It is suitable for various complex power operation scenarios, such as multi-step equipment maintenance or installation processes.

[0019] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0021] Figure 1 This is a flowchart illustrating the robotic arm control method based on a time-series vision operation manual provided in Embodiment 1 of the present invention. Figure 2This is a schematic diagram of a robotic arm control system based on a time-series vision operation manual provided in Embodiment 2 of the present invention. Figure 3 This is a schematic diagram of a computer device provided in Embodiment 3 of the present invention. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0024] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0025] Example 1: This implementation proposes a robotic arm control method based on a time-series visual operation manual, such as... Figure 1 As shown, the process includes the following: S1: Preprocess and identify the target in the acquired scene images to obtain the target identification results; S2: Based on the target recognition result, search in the predefined visual operation manual to obtain the time-series visual operation set of the robotic arm; S3: Perform path planning based on the time-series visual operation set, and perform motion control of the robotic arm based on the path planning results.

[0026] In this implementation, the creation of the predefined visual operation manual includes: First, the robotic arm is manually operated to complete the specified action, and then image data is acquired using a camera, represented as a pixel value matrix. ,in t It is a timestamp, representing a moment in the image sequence. t Image: (1); in, Dimensions H × W , H and W These are the height and width of the image, respectively. Indicates in The Middle i Okay, number j Pixel values ​​at the column; Secondly, the image sequence is sorted according to the timeline: (2); Where S represents the entire time series image set, T is the total number of time steps, and these images represent different time points in the robotic arm operation, each image... For a specific step in an operation, the timeline provides the continuity and sequence of the operation; Finally, a timeline guide for the creation process.

[0027] To ensure that the robotic arm can perform operations in sequence, we define a mapping function. , time t Mapped to the corresponding operation instruction set : (3); in, It is a mapping function from image to operation instructions (which can be based on the result of a visual recognition algorithm), and an operation instruction set. This includes information on position, angle, and task execution, along with the set of operational instructions at each moment. A visual operation manual for composing time series.

[0028] In this implementation, the robotic arm performs its tasks according to a predefined vision operation manual, specifically including: Image acquisition and key component identification: Image information is acquired through a camera, and the image recognition algorithm based on SAM analyzes the images to identify key components, tools, and their relative positions in the power equipment. Image acquisition: The camera captures images of the equipment site at a certain frequency, generating an image data matrix; Image preprocessing: Performing preprocessing operations such as filtering and noise reduction on the image to ensure that the image quality meets the recognition requirements; Target segmentation and recognition: The SAM model segments the targets in the image, divides the image into multiple regions, and analyzes each region to identify key components (such as transformers, switches, insulators, etc.), tools (such as wrenches, screwdrivers, etc.) in the power equipment, as well as their relative positions.

[0029] This information is represented in matrix form: (4); in Let n be the coordinates of key components and tools in the image, and n be the number of targets detected.

[0030] The system retrieves and extracts operational information from the visual operation manual. Based on the image information, it searches the visual operation manual to obtain operational information. After completing target recognition, the system matches the recognized information with the predefined visual operation manual to obtain the corresponding operation steps. The specific steps are as follows: Retrieval algorithm: Based on the target location and category, the system searches for the corresponding operation information in the predefined database or manual. This retrieval system uses hash mapping to quickly locate the required operation steps. Operation information extraction: Once a match is successful, the system extracts the operation information set. ,include: Target location: The three-dimensional coordinates of the electrical equipment component to be operated; Tool selection: Matching tool name and usage instructions; Operation sequence: The position and order of this component in the entire operation process.

[0031] Path planning and motion control: Based on the operational information parsed from the image, the robotic arm plans its path using a motion planning algorithm, setting precise motion trajectories and amplitudes. Path planning algorithm: The system uses an A* motion planning algorithm to calculate the optimal path from the robotic arm's current position to the target position, ensuring obstacle avoidance and optimizing motion time.

[0032] Trajectory Generation: Based on the planned path, a detailed motion trajectory is generated. Inverse kinematics is used to calculate the angles of each joint of the robotic arm, generating... : (5); in These are the three-dimensional coordinates of the target location. This is the set of angles for each joint.

[0033] Motion amplitude control: Set appropriate speed and acceleration parameters according to task requirements to ensure smooth and safe movement of the robotic arm.

[0034] Real-time image feedback and adaptive adjustment. The motion planning process incorporates real-time image feedback to ensure the robotic arm's every movement is precisely aligned with the target and adjusts to subtle environmental changes, preventing misoperations. During operation, the system monitors the robotic arm's movements using real-time image feedback and makes necessary adjustments based on environmental changes to maintain operational accuracy. Real-time image acquisition: The camera continuously captures images of the robotic arm's operations and compares them with the expected movements in the operation manual. Error detection: This involves calculating the error between the actual operation and the desired action. (6); in, and These are the actual location and the desired location, respectively.

[0035] Action adjustment: based on error value use PID The control system adaptively adjusts and updates operation commands in real time to ensure that the robotic arm can accurately align with the target every time it operates. (7); in, and To control the gain, the system's response speed and accuracy are adjusted.

[0036] Through the above process, the robotic arm can automatically perform precise operational tasks in complex power scenarios and make adaptive adjustments based on real-time changes to ensure safe and efficient operation.

[0037] Example 2: like Figure 2 As shown, this implementation provides a robotic arm control system based on a time-series vision operation manual, including: The target recognition unit is configured to: preprocess the acquired scene image and perform target recognition to obtain the target recognition result; The operation set retrieval unit is configured to: search in a predefined visual operation manual based on the target recognition result to obtain the time-series visual operation set of the robotic arm; The motion control unit is configured to: perform path planning based on the time-series visual operation set, and perform motion control of the robotic arm based on the path planning result.

[0038] The specific working process of each of the above units is described in Example 1, and will not be repeated here.

[0039] It is understood that the aforementioned units can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The aforementioned units are based on logical functional division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the system may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0040] According to another embodiment of this application, the system described in this embodiment, and the method of embodiment 1 of this application, can be constructed and the method of embodiment 1 of this application implemented by running a computer program (including program code) capable of performing the steps involved in the corresponding method described in embodiment 1 on a general-purpose computing device including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, loaded into the aforementioned computing device through the computer-readable recording medium, and run therein.

[0041] Example 3: like Figure 3 As shown, this implementation provides an electronic device including a processor 1001, a communication interface 1002, and a computer-readable storage medium 1003. The processor 1001, communication interface 1002, and computer-readable storage medium 1003 can be connected via a bus or other means.

[0042] The communication interface 1002 is used to receive and send data. The computer-readable storage medium 1003 can be stored in the memory of the electronic device. The computer-readable storage medium 1003 is used to store computer programs, which include program instructions. The processor 1001 is used to execute the program instructions stored in the computer-readable storage medium 1003.

[0043] The processor 1001 (or CPU (Central Processing Unit)) is the computing and control core of electronic devices. It is suitable for implementing one or more instructions, specifically for loading and executing one or more instructions to achieve corresponding methods or functions.

[0044] The processor 1001 is configured to perform the following process: The acquired scene images are preprocessed and target recognition is performed to obtain the target recognition results; Based on the target recognition results, a time-series visual operation set for the robotic arm is obtained by searching a predefined visual operation manual. Path planning is performed based on the time-series visual operation set, and the motion control of the robotic arm is performed based on the path planning results.

[0045] The specific working process is described in Example 1 and will not be repeated here.

[0046] Example 4: This implementation provides a computer-readable storage medium (Memory), which is a memory device in an electronic device used to store programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the electronic device and extended storage media supported by the electronic device. The computer-readable storage medium provides storage space that stores the processing system of the electronic device.

[0047] Furthermore, this storage space also contains one or more instructions suitable for loading and execution by the processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM memory or unstable memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0048] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to perform the following process: The acquired scene images are preprocessed and target recognition is performed to obtain the target recognition results; Based on the target recognition results, a time-series visual operation set for the robotic arm is obtained by searching a predefined visual operation manual. Path planning is performed based on the time-series visual operation set, and the motion control of the robotic arm is performed based on the path planning results.

[0049] The specific working process is described in Example 1 and will not be repeated here.

[0050] Example 5: This implementation provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the following process: The acquired scene images are preprocessed and target recognition is performed to obtain the target recognition results; Based on the target recognition results, a time-series visual operation set for the robotic arm is obtained by searching a predefined visual operation manual. Path planning is performed based on the time-series visual operation set, and the motion control of the robotic arm is performed based on the path planning results.

[0051] The specific working process is described in Example 1 and will not be repeated here.

[0052] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0053] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0054] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A robotic arm control method based on a time-series vision operation manual, characterized in that, Includes the following processes: The acquired scene images are preprocessed and target recognition is performed to obtain the target recognition results; Based on the target recognition results, a time-series visual operation set for the robotic arm is obtained by searching a predefined visual operation manual. The construction of a predefined visual operation manual includes: Images captured by cameras at various moments when the robotic arm is manually operated to complete a specified action are obtained; the image sequence is sorted according to the timeline to obtain an image sequence; the image sequence is mapped to the corresponding operation instruction set to obtain a predefined visual operation manual. Path planning is performed based on the time-series visual operation set, and the motion control of the robotic arm is performed based on the path planning result. The system captures images of the robotic arm in real time and compares them with the desired action images in the time-series visual operation set of the robotic arm. It calculates the error between the actual operation and the desired action. Based on the error value, it uses PID control for adaptive adjustment and updates the operation instructions in real time.

2. The robotic arm control method based on a time-series vision operation manual as described in claim 1, characterized in that, The target recognition results include: key components and tools in the scene image, as well as the relative positions between the key components and the tools.

3. The robotic arm control method based on a time-series vision operation manual as described in claim 1, characterized in that, A hash mapping is used to search in a predefined visual operation manual. When a match is found, the time-series visual operation set of the robotic arm is extracted. The time-series visual operation set includes: the three-dimensional coordinates of the power equipment component to be operated, the name and usage method of the matching tool, and the position and order of the key components in the entire operation process.

4. The robotic arm control method based on a time-series vision operation manual as described in claim 1, characterized in that, Based on motion planning algorithms, the optimal path from the current position of the robotic arm to the target position is calculated. Based on the optimal path, a detailed motion trajectory is generated, and inverse kinematics is used to calculate the angles of each joint of the robotic arm. According to the task requirements, the speed and acceleration parameters are set to make the movement of the robotic arm smooth and safe.

5. A robotic arm control system based on a time-series vision operation manual, characterized in that, include: The target recognition unit is configured to: preprocess the acquired scene image and perform target recognition to obtain the target recognition result; The operation set retrieval unit is configured to: search in a predefined visual operation manual based on the target recognition result to obtain the time-series visual operation set of the robotic arm; The construction of a predefined visual operation manual includes: Images captured by cameras at various moments when the robotic arm is manually operated to complete a specified action are obtained; the image sequence is sorted according to the timeline to obtain an image sequence; the image sequence is mapped to the corresponding operation instruction set to obtain a predefined visual operation manual. The motion control unit is configured to: perform path planning based on the time-series visual operation set, and perform motion control of the robotic arm based on the path planning result; The system captures images of the robotic arm in real time and compares them with the desired action images in the time-series visual operation set of the robotic arm. It calculates the error between the actual operation and the desired action. Based on the error value, it uses PID control for adaptive adjustment and updates the operation instructions in real time.

6. The robotic arm control system based on a time-series vision operation manual as described in claim 5, characterized in that, The target recognition results include: key components and tools in the scene image, as well as the relative positions between the key components and the tools.

7. The robotic arm control system based on a time-series vision operation manual as described in claim 5, characterized in that, A hash mapping is used to search in a predefined visual operation manual. When a match is found, the time-series visual operation set of the robotic arm is extracted. The time-series visual operation set includes: the three-dimensional coordinates of the power equipment component to be operated, the name and usage method of the matching tool, and the position and order of the key components in the entire operation process.

8. A computer device, characterized in that, include: Processor and computer-readable storage media; A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the robotic arm control method based on a time-series vision operation manual as described in any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1 to 4, which is a robotic arm control method based on a time-series vision operation manual.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the robotic arm control method based on a time-series vision operation manual as described in any one of claims 1 to 4.

Images