Dual-arm drawing robot teaching system and its drawing control method and device

Abstract

This invention provides a dual-arm painting robot teaching system and its painting control method and apparatus, relating to the field of dual-arm robots. The painting control method for this dual-arm painting robot includes: acquiring target painting information; calculating the painting trajectory of the first painting arm and the second painting arm using a deep learning algorithm based on the painting information collected by a machine vision system; acquiring detection information of the canvas and brush; determining whether the dual-arm painting robot can perform painting based on the detection information of the canvas and brush; and if the dual-arm painting robot can perform painting, controlling the left and right arms to coordinate and complete the painting task. The dual-arm painting robot teaching system and its painting control method and apparatus provided by this invention can improve painting efficiency.

Description

Technical Field

[0001] This invention relates to the field of dual-arm robots, and more specifically, to a dual-arm painting robot teaching system and its painting control method and apparatus. Background Technology

[0002] Humans have undergone millions of years of environmental adaptation, physical evolution, and intellectual development since their origins. The arms and fingers are a key distinguishing feature between humans and animals, exhibiting remarkable development and dexterity. A bi-armed robot is a humanoid robot with two functional robotic arms, typically used in industrial applications. However, because bi-armed robots mimic human arms, their applications extend beyond industry, encompassing other fields such as art instruction and demonstrations.

[0003] Currently, painting robots refer to humanoid robots capable of independently mimicking specific human behaviors. Structurally, they can be categorized into gantry-type painting robots and swing-arm-type painting robots. Swing-arm-type painting robots can be further divided into single-arm, dual-arm, and multi-arm types. Swing-arm-type painting robots offer high flexibility, strong programmability, and significant intelligence and creative capabilities, with great potential for future development. However, they have more degrees of freedom and more complex control. Compared to dual-arm robots, single-arm painting robots have a simpler structure, with each arm working independently and being easier to control. However, they are less efficient at drawing complex and intricate patterns, resulting in coarser strokes. Dual-arm painting robots simultaneously manage the control of both arms within a single control system, offering greater flexibility and adaptability. They are suitable for drawing complex and intricate patterns and have higher painting efficiency, but the coordinated movement planning and control of both arms is more challenging.

[0004] Image acquisition from drawing robots can be categorized into three methods: simple trajectory planning-based machine drawing, image processing-based machine drawing, and deep learning-based machine drawing. One method directly processes digital images to generate the trajectory of the robotic arm, producing the drawing. Another method uses machine vision technology to acquire images from a real-world environment, then processes these images to generate the drawing based on the execution trajectory. Visual image extraction techniques mostly rely on edge extraction to obtain the outline of a given face, converting it all into vector points for the robot to draw. This method is highly dependent on lighting conditions, does not perform line processing, has a long image processing time, produces unstable image quality, and is easily affected by external factors.

[0005] The main drawback of dual-arm robotic painting is the insufficient coordination of the two arms, making it difficult to demonstrate the advantages of dual-arm robots, and there are currently few reports on this technology. The biggest highlight of this invention, a dual-arm painting robot, is that it is based on a dual-arm robot, which is more in line with students' concept of "human" and is more flexible than a single-arm robot, enabling it to complete tasks that a single arm cannot. At the same time, it also utilizes a machine vision system and deep learning algorithms, making it more human-like and intelligent. Summary of the Invention

[0006] The present invention aims to, for example, provide a dual-arm drawing robot teaching system and its drawing control method and apparatus, which can improve drawing efficiency.

[0007] The embodiments of the present invention can be implemented as follows:

[0008] In a first aspect, embodiments of the present invention provide a painting control method for a dual-arm painting robot. Structurally, the dual-arm painting robot includes a first painting arm and a second painting arm. Hardware-wise, the dual-arm painting robot includes a detection and evaluation layer, a motion control layer, a communication layer, and a coordination and planning layer. The detection and evaluation layer includes a detection sensor system module, a vision system module, and an algorithm module. The motion control layer includes a left arm motion control module (i.e., the first painting arm) and a right arm motion control module (i.e., the second painting arm). The communication layer is a communication module and can use communication methods such as CAN / TCP / IP / WIFI. The coordination and planning layer includes a visual human-machine interface and the ability to connect and communicate with a tablet computer or computer. The painting control method includes:

[0009] Obtain target drawing information;

[0010] Based on the target drawing information, calculate the first drawing trajectory of the first drawing arm and the second drawing trajectory of the second drawing arm;

[0011] Obtain detection information for the drawing board and brushes;

[0012] Based on the detection information of the drawing board and brush, determine whether the dual-arm painting robot is capable of painting;

[0013] If the dual-arm painting robot is capable of painting, then the first painting arm is controlled to paint according to the first painting trajectory, and the second painting arm is controlled to paint according to the second painting trajectory.

[0014] Furthermore, in an optional embodiment, in the step of obtaining target painting information, the target painting information includes video data;

[0015] The step of calculating the first drawing trajectory of the first drawing arm and the second drawing trajectory of the second drawing arm based on the target drawing information includes:

[0016] Extract video keyframes from the video data;

[0017] The video data is preprocessed;

[0018] Feature extraction, feature filtering, and supplementation are performed on the preprocessed video data.

[0019] Furthermore, in an optional embodiment, in the step of extracting video keyframes from the video data, the frame difference method is used to find the video keyframes in the video data.

[0020] Furthermore, in an optional embodiment, in the step of preprocessing the video data, grayscale processing, median filtering, Gaussian smoothing, and binarization methods are used to preprocess the video data.

[0021] Further, in an optional embodiment, in the step of feature extraction from the preprocessed video data, the Canny algorithm is used to obtain facial contour and facial feature information from the video data, wherein the Canny algorithm includes the following steps:

[0022] The preprocessed video data is smoothed using a Gaussian filter.

[0023] The magnitude and direction of the gradient are calculated using the finite difference of the first-order partial derivatives.

[0024] Non-maximum suppression is applied to the gradient magnitude;

[0025] Edges are detected and connected using a dual threshold algorithm.

[0026] Furthermore, in an optional embodiment, in the step of feature filtering and supplementing the preprocessed video data, the Hough transform and the regular spline function in the RBF method are used to locate and supplement the identification feature lines, respectively.

[0027] Furthermore, in an optional embodiment, the method further includes a model reconstruction step, in which the model is reshaped based on SFM 3D reconstruction and Deep learning-based 3D reconstruction methods, and the model is optimized and decomposed into the first drawing trajectory of the first drawing arm and the second drawing trajectory of the second drawing arm using a Morse theory-based model segmentation algorithm.

[0028] Further, in an optional embodiment, in the step of optimizing and decomposing the model into the first drawing trajectory of the first drawing arm and the second drawing trajectory of the second drawing arm using the model segmentation algorithm based on Morse theory, a network model M is established using a camera; each vertex v is smoothed within a neighborhood of radius r; the key points of the network model M are calculated using the Morse function f according to the Morse Lemma, and a vertex set S is selected, and a segmentation boundary set B and a key segmentation point set P are created; a segmentation line set L is generated according to the segmentation line segments in the segmentation boundary set B, and a sub-model set Ms is generated according to the segmentation line set L; different robot processing trajectory generation methods are selected to generate the first drawing trajectory and the second drawing trajectory depending on whether the robot processing task is oriented towards a spatial curve or a spatial surface.

[0029] The painting control method for a dual-arm painting robot provided by this invention has the following beneficial effects: Based on the target painting information, a first painting trajectory of the first painting arm and a second painting trajectory of the second painting arm are calculated; based on the detection information of the canvas and brush, it is determined whether the dual-arm painting robot can perform painting; if the dual-arm painting robot can perform painting, the first painting arm is controlled to paint according to the first painting trajectory, and the second painting arm is controlled to paint according to the second painting trajectory. In other words, embodiments of this invention can calculate the first and second painting trajectories of the first and second painting arms based on the target painting information. These first and second painting trajectories together achieve the painting of the target painting information onto the canvas, thereby improving painting efficiency.

[0030] In a second aspect, embodiments of the present invention provide a painting control device for a dual-arm painting robot, the dual-arm painting robot comprising a first painting arm and a second painting arm, the painting control device comprising:

[0031] First acquisition module: used to acquire target painting information;

[0032] Calculation module: used to calculate the first drawing trajectory of the first drawing arm and the second drawing trajectory of the second drawing arm based on the target drawing information;

[0033] The second acquisition module is used to acquire detection information of the drawing board and brush.

[0034] Judgment module: used to determine whether the dual-arm painting robot can perform painting based on the detection information of the drawing board and the paintbrush;

[0035] Control module: If the dual-arm painting robot is capable of painting, control the first painting arm to paint according to the first painting trajectory, and control the second painting arm to paint according to the second painting trajectory.

[0036] Furthermore, in an optional embodiment, the computing module is also used for:

[0037] Extract video keyframes from the video data;

[0038] The video data is preprocessed;

[0039] Feature extraction, feature filtering, and supplementation are performed on the preprocessed video data.

[0040] Furthermore, in an optional embodiment, the video keyframes in the video data are identified using the frame difference method.

[0041] Furthermore, in an optional embodiment, the video data is preprocessed using grayscale processing, median filtering, Gaussian smoothing, and binarization methods.

[0042] Further, in an optional embodiment, the Canny algorithm is used to obtain facial contour and facial feature information from the video data, wherein the Canny algorithm includes the following steps:

[0043] The preprocessed video data is smoothed using a Gaussian filter.

[0044] The magnitude and direction of the gradient are calculated using the finite difference of the first-order partial derivatives.

[0045] Non-maximum suppression is applied to the gradient magnitude;

[0046] Edges are detected and connected using a dual threshold algorithm.

[0047] Furthermore, in an optional embodiment, the Hough transform and the regular spline function in the RBF method are used to locate and supplement the identification feature lines, respectively.

[0048] Furthermore, in an optional embodiment, the painting control device further includes a model reconstruction module, which is based on SFM 3D reconstruction and Deep learning-based 3D reconstruction model reshaping, and uses Morse theory-based model segmentation algorithm to optimize and decompose the model into the first painting trajectory of the first painting arm and the second painting trajectory of the second painting arm.

[0049] Further, in an optional embodiment, the model reconstruction module establishes a network model M using a camera; smooths each vertex v within a neighborhood of radius r; calculates key points of the network model M using the Morse function f according to Morse Lemma, selects a vertex set S, and creates a segmentation boundary set B and a key segmentation point set P; generates a segmentation line set L based on the segmentation line segments in the segmentation boundary set B, and generates a sub-model set Ms based on the segmentation line set L; and selects different robot processing trajectory generation methods to generate the first painting trajectory and the second painting trajectory depending on whether the robot processing task is oriented towards a spatial curve or a spatial surface.

[0050] The drawing control device for the dual-arm drawing robot provided in this embodiment calculates a first drawing trajectory of the first drawing arm and a second drawing trajectory of the second drawing arm based on target drawing information; it determines whether the dual-arm drawing robot can draw based on the detection information of the drawing board and the brush; if the dual-arm drawing robot can draw, it controls the first drawing arm to draw according to the first drawing trajectory and controls the second drawing arm to draw according to the second drawing trajectory. In other words, this embodiment of the invention can calculate the first and second drawing trajectories of the first and second drawing arms based on target drawing information. These first and second drawing trajectories together achieve the drawing of the target drawing information onto the drawing board, thereby improving drawing efficiency.

[0051] Thirdly, embodiments of the present invention provide a dual-arm painting robot education system, comprising: a dual-arm painting robot, the dual-arm painting robot including a first painting arm, a second painting arm and a controller, the first painting arm and the second painting arm being electrically connected to the controller, the controller storing an executable program, the program being executed to implement any of the aforementioned painting control methods; and a teaching resource library, the teaching resource library being communicatively connected to the controller.

[0052] The dual-arm drawing robot provided in this invention closely resembles the human form, aligning with students' concept of "human" and fostering a sense of familiarity, making it easy for students to accept and use in teaching. Furthermore, the movement of the two robotic arms is controlled and coordinated by a single control system, resulting in high flexibility, strong adaptability, and high drawing efficiency, capable of clearly drawing complex and intricate patterns. Moreover, the dual-arm drawing robot utilizes a machine vision system and deep learning algorithms to extract video footage, accurately identify and locate the outlines of patterns within the video frame, improving the accuracy and clarity of the drawing trajectory, and achieving anthropomorphic and intelligent robotic drawing. Attached Figure Description

[0053] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as limiting the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of the structure of the dual-arm painting robot according to a specific embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of the hardware module of the dual-arm painting robot according to a specific embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the structure of the dual-arm painting robot teaching system according to a specific embodiment of the present invention;

[0057] Figure 4 This is a schematic diagram of the painting control method of the dual-arm painting robot according to a specific embodiment of the present invention;

[0058] Figure 5 This is a flowchart of the painting control method for the dual-arm painting robot according to a specific embodiment of the present invention;

[0059] Figure 6 This is a diagram of a dual-connection-layer neural network model according to a specific embodiment of the present invention;

[0060] Figure 7 This is a schematic diagram of the single-arm control method according to a specific embodiment of the present invention;

[0061] Figure 8 This is a schematic diagram of the path planning method for a dual-arm painting robot according to a specific embodiment of the present invention.

[0062] Icons: 100, Dual-arm painting robot; 102, Controller; 110, Dual-arm robot; 111, First painting arm; 112, Second painting arm; 113, Control cabinet; 120, Operation cabinet; 130, Trajectory training platform. Detailed Implementation

[0063] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0064] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0065] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0066] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0067] It should be noted that, where there is no conflict, the features in the embodiments of the present invention can be combined with each other.

[0068] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0069] Please see Figure 1 This diagram illustrates the structure of a dual-arm painting robot 100 according to an embodiment of the present invention. The dual-arm painting robot 100 includes a dual-arm robot 110, an operation cabinet 120, and a trajectory training platform 130. The operation cabinet 120 can hold items. The trajectory training platform 130 is located above the operation cabinet 120. The operation cabinet 120 has casters below it, including two omnidirectional wheels and two directional wheels. The directional wheels are used for orientation and fixation, while the omnidirectional wheels allow for free movement, facilitating operation demonstrations in various locations. A drawing board support is located above the trajectory training platform 130, on which a drawing board is placed. Symmetrically arranged brush clamp supports are also present. The support consists of two supports with limiting grooves, allowing the brush clamp to be stably placed on the supports and preventing it from falling off. The brush clamp support has a brush clamp assembly, which includes a brush and a brush clamp. The brush clamp has a brush holding part below and a clamped part above. The clamped part is a cylindrical cylinder cut into four planes, with protrusions on the top of the planes to limit the movement and prevent the clamped part from slipping when gripped by the pneumatic gripper assembly. The bottom of the brush clamp is a square block that rests on the support of the brush clamp.

[0070] The dual-arm robot 110 includes a first painting arm 111, a second painting arm 112, and a control cabinet 113. The bottom of the control cabinet 113 is equipped with pulleys, including two omnidirectional wheels and two directional wheels. The control cabinet 113 is equipped with a control system for controlling the operation of the first painting arm 111 and the second painting arm 112 of the dual-arm robot 110. The ends of the first painting arm 111 and the second painting arm 112 are equipped with pneumatic gripper assemblies. The pneumatic gripper assembly includes a cylinder, two gripping plates, and a flange connecting plate. The flange connecting plate is connected to the lower end of the arm. The cylinder is used to push open and close the gripping plates to grasp the paintbrush clamp assembly. The gripping plates cooperate with the cross-section of the gripped part of the paintbrush clamp, so that when the cylinder pushes the gripping plates, it can firmly grasp the gripped part of the paintbrush clamp. Finally, the trajectory is taught on the drawing board.

[0071] Please see Figure 2 This invention provides a schematic diagram of the hardware modules of a dual-arm painting robot 100. The painting control method and device of the dual-arm painting robot 100 provided in this invention can be applied to the dual-arm painting robot 100. The dual-arm painting robot 100 includes a first painting arm 111 and a second painting arm 112. The aforementioned painting control method and device are used to control the running trajectory of the first painting arm 111 and the second painting arm 112, thereby achieving stable and efficient painting teaching. The painting control device includes at least one software function module that can be stored in the controller 102 in the form of software or firmware or embedded in the operating system (OS) of a server. The controller 102 is used to execute the executable module stored therein, such as the software function module and computer program included in the painting control device.

[0072] The controller 102 may be an integrated circuit chip with signal processing capabilities. The controller 102 described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor. The controller 102 can also be any conventional processor, etc.

[0073] The controller 102 is programmed with a drawing control program. When the controller 102 receives an execution instruction, it executes the drawing control program to implement the drawing control method described in this embodiment of the invention.

[0074] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0075] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0076] Please see Figure 2 The dual-arm painting robot 100 includes a detection and evaluation layer, a motion control layer, a communication layer, and a coordination and planning layer. The detection and evaluation layer includes a detection sensor system module, a vision system module, and an algorithm module. The motion control layer includes a left arm motion control module (i.e., the first painting arm 111) and a right arm motion control module (i.e., the second painting arm). The communication layer is a communication module that can use CAN / TCP / IP / WIFI and other communication methods. The coordination and planning layer includes a visual human-machine interface and can connect and communicate with a tablet computer or computer.

[0077] Please see Figure 3 In this embodiment, the dual-arm painting robot 100 can also be applied to a dual-arm painting robot 100 education system, which includes the dual-arm painting robot 100 and a teaching resource library. The software modules of the dual-arm painting robot 100 include a vision system, cabinet processing software, a multi-source multimodal perception model, an interactive experience module, a theory-practice integration implementation module, and the dual-arm robot 110 body (i.e., the hardware structure). The teaching resource library includes learning process control strategies, classroom teaching plans, control software implementation, robot execution control algorithms, and also includes teaching quality assessment and teaching process management.

[0078] Please see Figure 4 and Figure 5 This embodiment provides a painting control method for a dual-arm painting robot, including the following steps.

[0079] Step S100: Obtain target drawing information.

[0080] It should be noted that in step S100, the target painting information can be obtained through a video camera or by importing image or video files. This embodiment of the invention does not impose specific requirements or limitations on this.

[0081] Step S200: Calculate the first drawing trajectory of the first drawing arm and the second drawing trajectory of the second drawing arm based on the target drawing information.

[0082] In an optional embodiment, the target painting information includes video data in the step of obtaining target painting information. In step S100, the target painting information can be obtained by using a video camera or by importing video files.

[0083] In this embodiment, the step S200 of calculating the first drawing trajectory of the first drawing arm and the second drawing trajectory of the second drawing arm based on the target drawing information includes the following sub-steps S210, S220 and S230.

[0084] Sub-step S210: Extract video keyframes from the video data.

[0085] In this embodiment, in the step of extracting video keyframes from video data, the frame difference method is used to find the video keyframes in the video data.

[0086] Sub-step S220: Preprocess the video data.

[0087] In this embodiment, the video data is preprocessed using grayscale processing, median filtering, Gaussian smoothing, and binarization.

[0088] Sub-step S230: Perform feature extraction, feature filtering, and supplementation on the preprocessed video data.

[0089] In this embodiment, in the step of feature extraction from the preprocessed video data, the Canny algorithm is used to obtain facial contour and facial feature information from the video data. The Canny algorithm includes the following steps:

[0090] S231: Use a Gaussian filter to smooth the preprocessed video data;

[0091] S232: Calculate the magnitude and direction of the gradient using the finite difference of the first-order partial derivative;

[0092] S233: Non-maximum suppression of gradient magnitude;

[0093] S234: Detect and connect edges using a dual threshold algorithm.

[0094] In this embodiment, in the step of feature filtering and supplementation of preprocessed video data, the Hough transform and the regular spline function in the RBF method are used to locate and supplement the identification feature lines respectively.

[0095] In an optional embodiment, the painting control method further includes a model reconstruction step, in which the model is reshaped based on SFM 3D reconstruction and Deep learning-based 3D reconstruction methods, and the model is optimized and decomposed into a first painting trajectory of a first painting arm and a second painting trajectory of a second painting arm using a Morse theory model segmentation algorithm.

[0096] It should be noted that, in the embodiments of the present invention, the SFM algorithm can be implemented in the following manner.

[0097] Primarily based on the principle of multi-vision geometry, with the accumulation of CNNs in two-dimensional images, many CNN-based 2D depth estimation methods have achieved certain results. The CNN algorithm with two-layer convolutional neural networks is used for three-dimensional reconstruction.

[0098]

[0099] Figure 6 This is a diagram of the improved dual-connected-layer neural network model, whose two-dimensional convolution formula is defined as follows:

[0100] Where y(i,j) represents the output after 2D convolution, i, i,j are the corresponding coordinates in the image, k is the size of the convolution kernel, g(i,j) is the input of the convolution, and h(ki,kj) is a value in the convolution kernel.

[0101] It should be noted that, addressing the low recognition rate of traditional CNN neural networks when processing small datasets, the inventors of this application, through repeated adjustments to the model structure and combined with years of research in related fields, discovered that changes such as increasing the kernel size and the number of convolutional layers do not significantly improve the model's recognition performance and are prone to overfitting. By setting a random selection layer between two fully connected layers to randomly train the facial expression features after convolutional pooling, the model's recognition ability is significantly improved.

[0102] Please see Figure 7 and Figure 8Optionally, in this embodiment, in the step of optimizing and decomposing the model into the first drawing trajectory of the first drawing arm and the second drawing trajectory of the second drawing arm using the model segmentation algorithm based on Morse theory, a network model M is established using a camera; each vertex v is smoothed within a neighborhood of radius r; the key points of the network model M are calculated using the Morse function f according to the Morse Lemma, and a vertex set S is selected, and a segmentation boundary set B and a key segmentation point set P are created; a segmentation line set L is generated according to the segmentation line segments in the segmentation boundary set B, and a sub-model set Ms is generated according to the segmentation line set L; different robot processing trajectory generation methods are selected to generate the first drawing trajectory and the second drawing trajectory depending on whether the robot processing task is oriented towards a spatial curve or a spatial surface.

[0103] Step S300: Obtain detection information for the drawing board and brush.

[0104] It should be noted that step S300 mainly detects whether the drawing board and the pen are in the specified positions. When the drawing board and the pen are in the specified preset positions, the dual-arm drawing robot can draw; otherwise, the positions of the drawing board and the pen need to be adjusted.

[0105] Step S400: Based on the detection information of the drawing board and brush, determine whether the dual-arm painting robot can perform painting.

[0106] It should be noted that if the drawing board and pen are in their corresponding positions, the dual-arm drawing robot is deemed capable of drawing; if either the drawing board or pen is not in its corresponding position, the dual-arm robot is deemed unable to draw. When the drawing board or pen is not in its corresponding position, a reminder message can be sent to the user so that the user can address the issue promptly.

[0107] Step S500: If the dual-arm painting robot is capable of painting, control the first painting arm to paint according to the first painting trajectory, and control the second painting arm to paint according to the second painting trajectory.

[0108] It should be understood that, in this embodiment of the invention, a dual-arm painting robot is used, which has a first painting arm and a second painting arm. The first painting arm and the second painting arm can improve painting efficiency, thereby facilitating the painting display of the dual-arm painting robot.

[0109] The drawing control method for a dual-arm drawing robot provided in this embodiment of the invention involves: calculating a first drawing trajectory of the first drawing arm and a second drawing trajectory of the second drawing arm based on target drawing information; determining whether the dual-arm drawing robot can perform drawing based on the detection information of the drawing board and the brush; if the dual-arm drawing robot can perform drawing, controlling the first drawing arm to draw according to the first drawing trajectory and controlling the second drawing arm to draw according to the second drawing trajectory. In other words, this embodiment of the invention can calculate the first and second drawing trajectories of the first and second drawing arms based on the target drawing information, and these first and second drawing trajectories together achieve the drawing of the target drawing information onto the drawing board, thereby improving drawing efficiency.

[0110] This invention also provides a painting control device for a dual-arm painting robot. The dual-arm painting robot includes a first painting arm and a second painting arm. The painting control device includes a first acquisition module, a calculation module, a second acquisition module, a judgment module, and a control module.

[0111] In this embodiment of the invention, the first acquisition module is used to acquire target painting information.

[0112] Optionally, in this embodiment, the above-described step S100 is performed by the first acquisition module.

[0113] In this embodiment of the invention, the calculation module is used to calculate the first drawing trajectory of the first drawing arm and the second drawing trajectory of the second drawing arm based on the target drawing information.

[0114] Optionally, in this embodiment, the above-described step S200 is executed by the calculation module.

[0115] In this embodiment of the invention, the second acquisition module is used to acquire detection information of the drawing board and the brush.

[0116] Optionally, in this embodiment, step S300 described above is performed by the second acquisition module.

[0117] In this embodiment of the invention, the judgment module is used to determine whether the dual-arm painting robot can perform painting based on the detection information of the drawing board and the paintbrush.

[0118] Optionally, in this embodiment, the above-described step S400 is executed by the judgment module.

[0119] In this embodiment of the invention, the control module is used to control the first painting arm to paint according to the first painting trajectory and to control the second painting arm to paint according to the second painting trajectory if the dual-arm painting robot is capable of painting.

[0120] Optionally, in this embodiment, the above-described step S500 is executed by the control module.

[0121] The drawing control device for the dual-arm drawing robot provided in this embodiment calculates a first drawing trajectory of the first drawing arm and a second drawing trajectory of the second drawing arm based on target drawing information; it determines whether the dual-arm drawing robot can draw based on the detection information of the drawing board and the brush; if the dual-arm drawing robot can draw, it controls the first drawing arm to draw according to the first drawing trajectory and controls the second drawing arm to draw according to the second drawing trajectory. In other words, this embodiment of the invention can calculate the first and second drawing trajectories of the first and second drawing arms based on target drawing information. These first and second drawing trajectories together achieve the drawing of the target drawing information onto the drawing board, thereby improving drawing efficiency.

[0122] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0123] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0124] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0125] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0126] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A painting control method for a dual-arm painting robot, wherein the hardware of the dual-arm painting robot includes a detection and evaluation layer, a motion control layer, a communication layer, and a coordination and planning layer, wherein the detection and evaluation layer is used to detect and acquire data, the motion control layer is used to perform motion control on the dual-arm painting robot, and the coordination and planning layer is used to control the motion coordination of the dual-arm painting robot. The dual-arm painting robot includes a first painting arm and a second painting arm, characterized in that... The painting control method includes: acquiring target painting information; calculating a first painting trajectory of the first painting arm and a second painting trajectory of the second painting arm based on the target painting information; acquiring detection information of the drawing board and the brush; determining whether the dual-arm painting robot can paint based on the detection information of the drawing board and the brush; if the dual-arm painting robot can paint, controlling the first painting arm to paint according to the first painting trajectory and controlling the second painting arm to paint according to the second painting trajectory. In the step of acquiring target painting information, the target painting information includes video data; the step of calculating the first painting trajectory of the first painting arm and the second painting trajectory of the second painting arm based on the target painting information includes: extracting video keyframes from the video data; preprocessing the video data; and performing feature extraction, feature filtering, and supplementation on the preprocessed video data. The method further includes a model reconstruction step, in which the model is reshaped based on SFM 3D reconstruction and Deeplearning 3D reconstruction methods, and the model is optimized and decomposed into the first painting trajectory of the first painting arm and the second painting trajectory of the second painting arm using the Morse theory model segmentation algorithm. In the step of optimizing and decomposing the model into the first drawing trajectory of the first drawing arm and the second drawing trajectory of the second drawing arm using the model segmentation algorithm based on Morse theory, a network model M is established using a camera; each vertex v is smoothed within a neighborhood of radius r; the key points of the network model M are calculated using the Morse function f according to the Morse Lemma, and a vertex set S is selected, and a segmentation boundary set B and a key segmentation point set P are created; a segmentation line set L is generated according to the segmentation line segments in the segmentation boundary set B, and a sub-model set Ms is generated according to the segmentation line set L; different robot processing trajectory generation methods are selected according to whether the robot processing task is oriented towards spatial curves or spatial surfaces to generate the first drawing trajectory and the second drawing trajectory. In the step of feature extraction of the preprocessed video data, the Canny algorithm is used to obtain facial contour and facial feature information in the video data. The Canny algorithm includes the following steps: smoothing the preprocessed video data using a Gaussian filter; calculating the magnitude and direction of the gradient using the finite difference of the first-order partial derivative; performing non-maximum suppression on the gradient magnitude; and detecting and connecting edges using a double threshold algorithm.

2. The painting control method for the dual-arm painting robot according to claim 1, characterized in that, In the step of extracting video keyframes from the video data, the frame difference method is used to find the video keyframes in the video data.

3. The painting control method for a dual-arm painting robot according to claim 1, characterized in that, In the step of preprocessing the video data, grayscale processing, median filtering, Gaussian smoothing, and binarization methods are used to preprocess the video data.

4. The painting control method for a dual-arm painting robot according to claim 1, characterized in that, In the step of feature filtering and supplementation of the preprocessed video data, the Hough transform and the regular spline function in the RBF method are used to locate and supplement the identification feature lines respectively.

5. A dual-arm drawing robot education system, characterized in that, include: A dual-arm painting robot, comprising a first painting arm, a second painting arm, and a controller, wherein both the first and second painting arms are electrically connected to the controller, and the controller stores an executable program that, when executed, implements the painting control method as described in any one of claims 1-4; and a teaching resource library, communicatively connected to the controller, comprising a learning process control strategy module, a classroom teaching plan module, a control software implementation module, a robot execution control algorithm module, a teaching quality evaluation module, and a teaching process management module.

Images