Workpiece spraying method and device applied to spraying robot and storage medium

Abstract

The invention discloses a workpiece spraying method and device applied to a spraying robot and a storage medium. The method comprises the steps that a sprayed workpiece image and a to-be-classified demonstration video are acquired; the spraying workpiece image is input into a target workpiece detection model, a target workpiece detection result is obtained, and the target workpiece detection result comprises workpiece bounding box coordinates and workpiece types; generating an object model according to the target workpiece detection result; inputting the demonstration videos to be classified into a spraying video classification model to obtain a spraying video classification result; generating a target multi-modal prompt according to a multi-modal prompt template, the object model and the spraying video classification result; inputting the target multi-mode prompt into a motion trail prediction model to obtain a robot motion trail; and according to the motion trail of the robot, workpiece spraying is conducted. Workpiece spraying is achieved, and the spraying precision and applicability are improved. The spraying robot can be widely applied to the technical field of spraying robots.

Description

Technical Field

[0001] The present invention relates to the technical field of spraying robots, and in particular to a workpiece spraying method, device and storage medium applied to a spraying robot. Background Art

[0002] The spraying robot needs to spray the spraying object according to the preset requirements. Traditional spraying robot systems mostly use predetermined paths, fixed speeds and spraying postures to perform tasks. However, they perform poorly when the paths and speeds need to be adjusted in real time for different environmental changes and complex spraying requirements, and the spraying accuracy is low. At the same time, spraying robots combined with deep learning usually require a large amount of training data, and have a single function, making it difficult to adapt to diverse industrial needs and having low applicability.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The embodiments of the present invention provide a workpiece spraying method, device and storage medium applied to a spraying robot, which effectively improves the spraying accuracy and applicability.

[0005] On the one hand, an embodiment of the present invention provides a workpiece spraying method applied to a spraying robot, comprising the following steps:

[0006] Acquire spraying workpiece images and demonstration videos to be classified;

[0007] Inputting the sprayed workpiece image into a target workpiece detection model to obtain a target workpiece detection result, wherein the target workpiece detection result includes a workpiece boundary box coordinate and a workpiece category;

[0008] generating an object model according to the target workpiece detection result;

[0009] Inputting the demonstration video to be classified into a spraying video classification model to obtain a spraying video classification result;

[0010] Generate a target multimodal prompt according to the multimodal prompt template, the object model and the spraying video classification result;

[0011] Inputting the target multimodal prompt into a motion trajectory prediction model to obtain the robot motion trajectory;

[0012] The workpiece is sprayed according to the robot motion trajectory.

[0013] In some embodiments, the target workpiece detection model is obtained by the following steps:

[0014] Acquire the depth image and three-channel color image of the sprayed workpiece;

[0015] Fusing the depth image and the three-channel color image to obtain a fused image;

[0016] Calculating pixel coordinate height according to the depth image;

[0017] After the fused image is annotated with an image label, the fused image and the pixel coordinate height are input into a preset target detection model so that the preset deep learning model is trained to obtain the target workpiece detection model.

[0018] In some embodiments, the spraying video classification model is obtained by the following steps:

[0019] Get a spraying demonstration video;

[0020] After the spraying demonstration video is labeled with task category labels, format preprocessing is performed on the spraying demonstration video, wherein the task category labels include moving objects, flipping objects, rotating objects, and spraying objects;

[0021] Divide each frame of the spraying demonstration video after format preprocessing to obtain a plurality of image blocks;

[0022] According to the self-attention mechanism and the category probability distribution objective function, the plurality of image blocks are input into a preset classification model so that the preset classification model is trained to obtain the spraying video classification model.

[0023] In some embodiments, the motion trajectory prediction model is obtained by the following steps:

[0024] Obtain multimodal prompts and motion trajectory training data to be trained;

[0025] According to the path optimization cost function and the path prediction function, the multimodal prompt to be trained and the motion trajectory training data are input into a preset multimodal robot agent model so that the preset multimodal robot agent model is trained to obtain the motion trajectory prediction model.

[0026] In some embodiments, spraying the workpiece according to the robot motion trajectory includes:

[0027] Converting the robot motion trajectory into motion instruction sequence text;

[0028] Sending the motion instruction sequence text to the spraying robot;

[0029] The spraying robot is used to convert the motion instruction sequence text into a parameter list, and generate a motor control signal through a controller according to the parameter list;

[0030] The motor control signal is used to control the spraying robot to perform a workpiece spraying operation.

[0031] In some embodiments, the expression of the self-attention mechanism is:

[0032]

[0033] Where Attention is the self-attention mechanism, softmax is the normalized exponential function, Q is the query matrix, K is the key matrix, V is the value matrix, and d k is the dimension of the key vector.

[0034] In some embodiments, the expression of the category probability distribution objective function is:

[0035]

[0036] In the formula, is the category probability distribution objective function, N is the number of task categories, P(c|x) is the probability of task category c under input x, c is the task category, and x is the video sample.

[0037] On the other hand, an embodiment of the present invention provides a workpiece spraying device applied to a spraying robot, comprising:

[0038] The first module is used to obtain the spray workpiece image and the demonstration video to be classified;

[0039] The second module is used to input the sprayed workpiece image into a target workpiece detection model to obtain a target workpiece detection result, wherein the target workpiece detection result includes a workpiece boundary box coordinate and a workpiece category;

[0040] A third module is used to generate an object model according to the target workpiece detection result;

[0041] The fourth module is used to input the demonstration video to be classified into the spraying video classification model to obtain the spraying video classification result;

[0042] A fifth module is used to generate a target multimodal prompt according to the multimodal prompt template, the object model and the spraying video classification result;

[0043] The sixth module is used to input the target multimodal prompt into the motion trajectory prediction model to obtain the robot motion trajectory;

[0044] The seventh module is used to spray the workpiece according to the movement trajectory of the robot.

[0045] In another aspect, an embodiment of the present invention provides a computer device, comprising:

[0046] at least one processor;

[0047] at least one memory for storing at least one program;

[0048] When the at least one program is executed by the at least one processor, the at least one processor implements the described method.

[0049] On the other hand, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0050] The beneficial effects of the present invention are as follows:

[0051] The embodiment of the present invention first obtains a spray workpiece image and a demonstration video to be classified, inputs the spray workpiece image into a target workpiece detection model to obtain a target workpiece detection result, and generates an object model based on the target workpiece detection result, and then inputs the demonstration video to be classified into a spray video classification model to obtain a spray video classification result, and generates a target multimodal prompt based on a multimodal prompt template, an object model and a spray video classification result, and then inputs the target multimodal prompt into a motion trajectory prediction model to obtain a robot motion trajectory, and finally performs workpiece spraying based on the robot motion trajectory, so that workpiece spraying can be achieved by predicting the motion trajectory, thereby improving the spraying accuracy and applicability.

[0052] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 This is a flow chart of a workpiece spraying method applied to a spraying robot according to an embodiment of the present invention;

[0055] Figure 2 This is a schematic structural diagram of a workpiece spraying device applied to a spraying robot according to an embodiment of the present invention;

[0056] Figure 3 The figure is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0058] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0059] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0061] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0062] Spraying: refers to a coating method that uses a spray gun or a disc atomizer to disperse into uniform and fine droplets with the help of pressure or centrifugal force and apply them to the surface of the object to be coated. It can be divided into air spraying, airless spraying, electrostatic spraying and various derivative methods of the above basic spraying forms, such as high-flow low-pressure atomization spraying, thermal spraying, automatic spraying, multi-group spraying, etc.

[0063] In the related art, the current spray robot system mostly uses a predetermined path, fixed speed and spraying posture to perform tasks. This method performs well in high repeatability and high precision tasks, but often performs poorly in complex scenarios where the path and speed need to be adjusted in real time. In addition, if the existing spray robot adopts deep learning, it usually requires a large amount of training data and cost, and the single function is difficult to adapt to the diverse industrial needs. For environmental changes and complex spraying requirements, the existing technology usually requires manual intervention or reprogramming or training. And under the current technical background, the improvement direction of the spray robot path planning is mainly focused on using computer vision and machine learning technology to improve the spraying quality and efficiency through accurate detection of workpiece surface features and path optimization based on historical data. However, these methods still have shortcomings when dealing with specific workpiece shapes and spraying requirements (for example, spraying a cube but only five faces need to be sprayed). At the same time, the existing spray robot system is difficult to adapt to the ever-changing production environment, and cannot modify the spray path in time according to different requirements and goals, and lacks flexibility. The existing spray robot system requires a large amount of training data and high computing resources, which increases the application cost, high cost and high data demand.

[0064] In view of this, this embodiment generates multimodal prompts and forms a new path planning method by combining the position information of the sprayed workpiece and the demonstration video of manual spraying, and then executes the robot movement and spraying operations through the planned path to meet the special spraying needs in more possible scenarios.

[0065] A workpiece spraying method for a spraying robot provided in an embodiment of the present application relates to the technical field of spraying robots. A workpiece spraying method for a spraying robot provided in an embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The server can also be a node server in a blockchain network; the software can be an application that implements a workpiece spraying method for a spraying robot, etc., but is not limited to the above forms.

[0066] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0067] The following is a detailed explanation of the embodiments of the present application in conjunction with the accompanying drawings:

[0068] Figure 1 is an optional flow chart of a workpiece spraying method applied to a spraying robot provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0069] Step S101, obtaining a spraying workpiece image and a demonstration video to be classified;

[0070] Step S102, inputting the sprayed workpiece image into the target workpiece detection model to obtain the target workpiece detection result, which includes the workpiece boundary box coordinates and the workpiece category;

[0071] Step S103, generating an object model according to the target workpiece detection result;

[0072] Step S104, inputting the demonstration video to be classified into the spraying video classification model to obtain the spraying video classification result;

[0073] Step S105, generating a target multimodal prompt according to the multimodal prompt template, the object model and the spraying video classification result;

[0074] Step S106, inputting the target multimodal prompt into the motion trajectory prediction model to obtain the robot motion trajectory;

[0075] Step S107: spraying the workpiece according to the robot's motion trajectory.

[0076] Steps S101 to S107 shown in the embodiment of the present application realize workpiece spraying and improve spraying accuracy and applicability.

[0077] In step S101 of some embodiments, a spray workpiece image of the spray object can be obtained by a depth camera (such as Realsense), and a demonstration video to be classified can be obtained by the same depth camera. The spray workpiece image and the demonstration video to be classified can also be obtained by other methods, not limited thereto.

[0078] In some embodiments, in step S102, the sprayed workpiece image can be input into the target workpiece detection model to obtain the target workpiece detection result, wherein the target workpiece detection result includes the workpiece bounding box coordinates and the workpiece category. It can be understood that by classifying the sprayed workpiece according to the target workpiece detection model, the type information of the sprayed workpiece can be obtained, including the workpiece bounding box coordinates and the workpiece category.

[0079] In some embodiments, the target artifact detection model may be obtained by the following steps:

[0080] Obtain depth images and three-channel color images of the sprayed workpiece;

[0081] The depth image and the three-channel color image are fused to obtain a fused image;

[0082] According to the depth image, calculate the pixel coordinate height;

[0083] After the fused image is annotated with an image label, the fused image and the pixel coordinate height are input into a preset target detection model so that the preset deep learning model is trained to obtain a target workpiece detection model.

[0084] In some embodiments, the depth image and three-channel color image of the sprayed workpiece can be obtained first. For example, a depth camera (such as Realsense) can be used to obtain the depth information and position information of the sprayed object, and the depth image and the three-channel color image (i.e., RGB image) can be obtained. Then the depth image and the three-channel color image are fused to obtain a fused image to obtain richer object information, such as surface material and three-dimensional shape features. According to the depth image, the pixel coordinate height is calculated. For example, the depth of an object or position on the image can be obtained according to the coordinate point corresponding to the image, and the lens height is subtracted from the depth to obtain the actual height of the current pixel coordinate, i.e., the pixel coordinate height. Finally, after the fused image is annotated with the image label, the fused image and the pixel coordinate height are input into the preset target detection model so that the preset deep learning model is trained to obtain the target workpiece detection model. For example, the preset target detection model can be set to the YOLOv8 target detection model. After collecting the object image that needs to be sprayed, each sprayed workpiece is labeled, and the training method provided by ultralytics is used on the computer and trained on pytorch. The goal is to be able to mark all objects and their types with boxes for a given scene image of a sprayed workpiece. The output of the YOLOv8 model includes the coordinates (x, y, w, h) and category confidence of each bounding box, thereby calculating the object category probability. The calculation formula for the object category probability is: P(c|box)=σ(t c )×P o , where P(c|box) is the object category probability, σ(t c ) is the category confidence, P o is the probability of the existence of the object. Furthermore, the continuous probability information can be converted into discrete specific object category information and its existence information by manually setting the category confidence and probability threshold, and the existence information includes yes or no.

[0085] In some embodiments, in step S103, an object model can be generated according to the target workpiece detection result. After the spraying workpiece is classified, more features can be extracted according to the type information (including the workpiece bounding box coordinates and the workpiece category). For example, after classification, it may be a wooden block, then the wood texture and glossiness of the object surface are obtained to generate an object model (obj model) closer to the actual situation. An obj model of an object can be created for each spraying workpiece for use in a simulation environment and generated in a pybullet simulation environment. This embodiment combines depth information and RGB images, filters out irrelevant background objects, and only retains information about foreground objects related to the spraying task, which can reduce interference with model processing.

[0086] In some embodiments, in step S104, the demonstration video to be classified can be input into the spraying video classification model to obtain the spraying video classification result. Exemplarily, the spraying video data demonstrated by humans (i.e., the demonstration video to be classified) can be classified into several different types of subtasks through the video classification model ViT (Vision Transformer), and the steps (tasks) performed by the workers in the video can be obtained, so as to generate multimodal prompts according to different task types later. More importantly, for a workpiece to be sprayed, the demonstration video to be classified of the task can be input into the spraying video classification model by file uploading or on-site demonstration recording, and after analysis and prediction by the spraying video classification model, the task classification label demonstrated by the user is output.

[0087] In some embodiments, the spraying video classification model is obtained by the following steps:

[0088] Get a spraying demonstration video;

[0089] After the spraying demonstration video is labeled with task category labels, the spraying demonstration video is format preprocessed. The task category labels include moving objects, flipping objects, rotating objects, and spraying objects.

[0090] Each frame of the spraying demonstration video after format preprocessing is divided into several image blocks;

[0091] According to the self-attention mechanism and the category probability distribution objective function, several image blocks are input into the preset classification model so that the preset classification model is trained to obtain the spraying video classification model.

[0092] In some embodiments, a spraying demonstration video can be obtained first. For example, a video of a worker's demonstration can be recorded according to different types of tasks. When acquiring video data, it should be ensured that the acquisition method is the same as that of image data (i.e., the depth image and three-channel color image of the sprayed workpiece), and the same camera settings (such as resolution, height, brightness, etc.) are used to ensure that the objects in the video can be accurately distinguished. Then after the spraying demonstration video is annotated with the task category label, the spraying demonstration video is formatted and preprocessed, wherein the task category label includes moving objects, flipping objects, rotating objects, and spraying objects. Optionally, in the process of annotating the task category label for the spraying demonstration video, the task category label is a number, and according to different task category labels (numbers), the tasks to be performed are determined. For example, the classification of tasks can include moving objects, flipping objects, rotating objects, and spraying objects. Among them, moving objects is used to characterize moving objects from one place to another; flipping objects is used to characterize flipping objects; rotating objects (x°) is used to characterize rotating objects by x°; spraying objects is used to characterize spraying one or more specified faces of objects, which can be "spraying the top of the object", "spraying the left side of the object", "spraying all faces of the object (except the bottom because it is on the table)" in the prompt. If there are special needs, training can be performed through different task classifications and data. Assuming that there are N (N is 4 in this embodiment) task categories, the goal of the model classifier is to maximize the probability distribution of the category. Optionally, the expression of the category probability distribution objective function is: In the formula, is the category probability distribution objective function, N is the number of task categories, P(c|x) is the probability of task category c under input x, c is the task category, and x is the video sample.

[0093] Then, each frame image in the spraying demonstration video after format preprocessing is divided into several image blocks. For example, the features can be analyzed by the encoder (Encoder), and each frame image of the video can be divided into n*n image blocks. For each image block, the attention between it and the image blocks at different times but the same position, and the attention between the image blocks at the same time but different positions are calculated through the attention mechanism. Finally, according to the self-attention mechanism and the category probability distribution objective function, several image blocks are input into the preset classification model so that the preset classification model is trained to obtain the spraying video classification model. For example, it can be trained on the pytorch platform, and the goal is to be able to distinguish what task is done for the input spraying demonstration video. The video classification model ViT uses the Transformer architecture to process images. The model first divides the image into patches of fixed size, and then flattens and embeds these patches into a high-dimensional space. Through the self-attention mechanism of the Transformer, the model can capture the relationship between remote pixels in the image. Optionally, the expression of the self-attention mechanism of ViT is: In the formula, Attention is the self-attention mechanism, softmax is the normalized exponential function, Q is the query matrix, K is the key matrix, V is the value matrix, and d k is the dimension of the key vector.

[0094] In some embodiments, in step S105, a target multimodal prompt can be generated based on the multimodal prompt template, the object model and the spraying video classification result. For each task category, a multimodal prompt template can be pre-defined, such as "spray {object A} on top", where "{object A}" is a vacant space in the image modality waiting to be filled. After obtaining the object model of the spray workpiece image target, fill it into the multimodal prompt template, and finally obtain a complete target multimodal prompt for the current task. For example, "spray all sides of object 'A', turn object 'A' over, and spray the top of object 'A'". It can be understood that the three processes of the example complete the task of spraying all surfaces of an object, which includes three tasks, which are separated by commas in the prompt. In the multimodal prompt, the single quotes refer to the object, and 'A' is an image of an object, so as to combine text and image to clearly indicate which object is to be operated.

[0095] In some embodiments, in step S106, the target multimodal prompt can be input into the motion trajectory prediction model to obtain the robot motion trajectory. Exemplarily, a VIMA (Visuomotor Attention Agent) multimodal robot agent based on multimodal prompts can be pre-trained, and the multimodal robot agent can output a robot control instruction sequence (i.e., robot motion trajectory) after inputting a string of multimodal prompts that intersect text and images. Optionally, VIMA can be used to calculate the optimal spraying path in real time to obtain the robot's motion instruction sequence. More importantly, the target multimodal prompt is input into VIMA, and the optimal spraying path is automatically inferred and calculated. Directly act on the simulation environment, after successful spraying in the simulation environment, it can be connected to the real-world robot through socket communication in the future, and the predicted robot instruction sequence can be applied to the real-world robot.

[0096] In some embodiments, the motion trajectory prediction model is obtained by the following steps:

[0097] Obtain multimodal prompts and motion trajectory training data to be trained;

[0098] According to the path optimization cost function and the path prediction function, the multimodal prompts to be trained and the motion trajectory training data are input into the preset multimodal robot agent model, so that the preset multimodal robot agent model is trained to obtain the motion trajectory prediction model.

[0099] In some embodiments, the multimodal prompts to be trained and the motion trajectory training data can be obtained first. Exemplarily, the training data can be generated by the official data generator, which can generate a large number of multimodal prompts to be trained according to the required tasks, as well as the robot motion sequence (i.e., motion trajectory training data) for the multimodal prompts to be trained. Then, according to the path optimization cost function and the path prediction function, the multimodal prompts to be trained and the motion trajectory training data are input into the preset multimodal robot agent model so that the preset multimodal robot agent model is trained to obtain the motion trajectory prediction model. Exemplarily, for the generated data (the pre-trained model is completed), after writing the running example, it is referenced in the form of a function in the form of an external import package. That is, when used, write the automation code, transfer the target multimodal prompt as a parameter to the prediction function, and finally obtain the predicted robot action sequence. Optionally, in the multimodal robot agent VIMA, for each task, a target multimodal prompt combining a text description and an image is provided, and then a large amount of training data (the robot's motion trajectory) is generated by the Oracle method officially provided by VIMA. The two are combined through training. The goal is to infer the robot's motion trajectory by inputting the target multimodal prompt with both text and image. Its multimodal fusion can be achieved through the self-attention mechanism, in which the input text and image are processed into a query matrix, a key matrix, and a value matrix. The path prediction function can be defined as: H = MultiHead(Q text ,K img ,V img ), where H is the prediction path, MultiHead is the multi-head self-attention mechanism, and Q text is the query matrix of text input, K 1mg is the key matrix of the image input, V img A matrix of values ​​for the image input. More generally, the goal of path optimization is to minimize the cost function of the robot's task execution, which may include time, path length, energy consumption, and so on. Through experiments in a simulation environment, the model can learn the optimal motion path. The real-time adjustment part involves feedback based on the actual environment. For example, if there are multiple objects in the scene, an object will be placed in a certain position after spraying. At this time, the robot needs to automatically plan the placement position and avoid these placed objects in the next movement. If objects need to be stacked, the placement height can be obtained in real time through a depth camera, dynamically calculated, and then the path planning is dynamically updated. In path planning, the path optimization cost function can be set as the loss function of the optimization objective: Among them, J(θ) is the path optimization cost function, θ is the strategy parameter, τ is the trajectory, and r t is the reward at time t, T is the total number of times, For strategy πθ The expectation of the next trajectory τ.

[0100] In some embodiments, in step S107, spraying the workpiece is performed according to the robot motion trajectory, which may include but is not limited to the following steps:

[0101] Convert the robot motion trajectory into motion instruction sequence text;

[0102] Send the motion instruction sequence text to the spraying robot;

[0103] The spraying robot is used to convert the motion instruction sequence text into a parameter list, and generate a motor control signal through the controller according to the parameter list;

[0104] The motor control signal is used to control the spraying robot to perform workpiece spraying operations.

[0105] In some embodiments, the robot motion trajectory can be converted into a motion instruction sequence text first, and the motion instruction sequence text is sent to the spraying robot. Exemplarily, for the predicted robot motion trajectory, each step thereof is converted into the form of information text (i.e., motion instruction sequence text) by a function, such as in the function form 'movej a, b', a is the joint to be moved, b is the position to be moved to, b can include multiple parameters, and a has only one parameter; in the form of information text, such as '0, 0.1258, 0.4172', it represents moving joint 0 to the position of 0.1258, 0.4172, and then the motion instruction sequence text is sent to the spraying robot via socket communication. After the spraying robot receives the motion instruction sequence text, the motion instruction sequence text can be converted into a textual parameter list by the spraying robot, and according to the parameter list, a motor control signal is generated by a controller. Wherein, the parameter list will be used in a pre-specified robot motion function (such as movej) to complete the control code of the spraying robot, and the control code can be run later and the motion and spraying operation of the spraying robot can be controlled. At the same time, the spraying operation can be performed by a controller (such as PID controller, LQR, etc.). The task of the controller is to convert the high-level motion instruction sequence text into a low-level motor control signal, which can be used to control the spraying robot to perform the workpiece spraying operation. Optionally, the output u(t) of the PID controller is determined by the error e(t), which can be expressed as: Where u(t) is the output of the controller, K p is the ratio, K i is the integral, K d is the differential gain, e(t) is the error between the target position and the current actual position, and t is the time.

[0106] The beneficial effects of implementing the embodiments of the present invention include: the embodiments of the present invention first obtain the spray workpiece image and the demonstration video to be classified, input the spray workpiece image into the target workpiece detection model, obtain the target workpiece detection result, and generate the object model according to the target workpiece detection result, and then input the demonstration video to be classified into the spray video classification model to obtain the spray video classification result, and generate the target multimodal prompt according to the multimodal prompt template, the object model and the spray video classification result, and then input the target multimodal prompt into the motion trajectory prediction model to obtain the robot motion trajectory, and finally according to the robot motion trajectory, the workpiece is sprayed, so that the workpiece spraying can be achieved by predicting the motion trajectory, thereby improving the spraying accuracy and applicability. At the same time, this embodiment can accurately identify the spray workpiece object, accurately classify the spraying task, and infer the optimal spraying path in real time according to the target multimodal prompt. This embodiment can adapt to environmental changes in real time, dynamically adjust the spraying path and parameters, and has high flexibility; and reduces manual intervention, improves production efficiency and automation.

[0107] like Figure 2 As shown, an embodiment of the present invention further provides a workpiece spraying device applied to a spraying robot, comprising:

[0108] The first module 801 is used to obtain the spray workpiece image and the demonstration video to be classified;

[0109] The second module 802 is used to input the sprayed workpiece image into the target workpiece detection model to obtain the target workpiece detection result, which includes the workpiece boundary box coordinates and the workpiece category;

[0110] The third module 803 is used to generate an object model according to the target workpiece detection result;

[0111] The fourth module 804 is used to input the demonstration video to be classified into the spraying video classification model to obtain the spraying video classification result;

[0112] The fifth module 805 is used to generate a target multimodal prompt according to the multimodal prompt template, the object model and the spraying video classification result;

[0113] The sixth module 806 is used to input the target multimodal prompt into the motion trajectory prediction model to obtain the robot motion trajectory;

[0114] The seventh module 807 is used to spray the workpiece according to the robot's motion trajectory.

[0115] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0116] like Figure 3 As shown, an embodiment of the present invention further provides a computer device, including:

[0117] at least one processor 901;

[0118] At least one memory 902, used to store at least one program;

[0119] When at least one program is executed by at least one processor, the at least one processor implements Figure 1 The method shown.

[0120] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0121] The embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, Figure 1 The method shown.

[0122] The contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0123] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A workpiece spraying method applied to a spraying robot, characterized in that: The following steps are involved: Acquire spray workpiece images and demonstration videos to be classified; Inputting the sprayed workpiece image into a target workpiece detection model to obtain a target workpiece detection result, wherein the target workpiece detection result includes a workpiece boundary box coordinate and a workpiece category; generating an object model according to the target workpiece detection result; Inputting the demonstration video to be classified into a spraying video classification model to obtain a spraying video classification result; Generate a target multimodal prompt according to the multimodal prompt template, the object model and the spraying video classification result; Inputting the target multimodal prompt into a motion trajectory prediction model to obtain the robot motion trajectory; The workpiece is sprayed according to the robot motion trajectory.

2. The method according to claim 1, characterized in that The target workpiece detection model is obtained by the following steps: Obtain depth images and three-channel color images of the sprayed workpiece; Fusing the depth image and the three-channel color image to obtain a fused image; Calculating pixel coordinate height according to the depth image; After the fused image is annotated with an image label, the fused image and the pixel coordinate height are input into a preset target detection model so that the preset deep learning model is trained to obtain the target workpiece detection model.

3. The method according to claim 1, characterized in that The spraying video classification model is obtained by the following steps: Get a spraying demonstration video; After the spraying demonstration video is labeled with task category labels, format preprocessing is performed on the spraying demonstration video, wherein the task category labels include moving objects, flipping objects, rotating objects, and spraying objects; Divide each frame of the spraying demonstration video after format preprocessing to obtain a plurality of image blocks; According to the self-attention mechanism and the category probability distribution objective function, the plurality of image blocks are input into a preset classification model so that the preset classification model is trained to obtain the spraying video classification model.

4. The method according to claim 1, characterized in that The motion trajectory prediction model is obtained by the following steps: Obtain multimodal prompts and motion trajectory training data to be trained; According to the path optimization cost function and the path prediction function, the multimodal prompt to be trained and the motion trajectory training data are input into a preset multimodal robot agent model so that the preset multimodal robot agent model is trained to obtain the motion trajectory prediction model.

5. The method according to claim 1, characterized in that The process of spraying the workpiece according to the robot motion trajectory comprises: Converting the robot motion trajectory into motion instruction sequence text; Sending the motion instruction sequence text to the spraying robot; The spraying robot is used to convert the motion instruction sequence text into a parameter list, and generate a motor control signal through a controller according to the parameter list; The motor control signal is used to control the spraying robot to perform a workpiece spraying operation.

6. The method according to claim 3, characterized in that The expression of the self-attention mechanism is: Where Attention is the self-attention mechanism, softmax is the normalized exponential function, Q is the query matrix, K is the key matrix, V is the value matrix, and d k is the dimension of the key vector.

7. The method according to claim 3, characterized in that The expression of the category probability distribution objective function is: In the formula, is the category probability distribution objective function, N is the number of task categories, P(c|x) is the probability of task category c under input x, c is the task category, and x is the video sample.

8. A workpiece spraying device applied to a spraying robot, characterized in that: include: The first module is used to obtain the spray workpiece image and the demonstration video to be classified; The second module is used to input the sprayed workpiece image into a target workpiece detection model to obtain a target workpiece detection result, wherein the target workpiece detection result includes a workpiece boundary box coordinate and a workpiece category; A third module is used to generate an object model according to the target workpiece detection result; The fourth module is used to input the demonstration video to be classified into the spraying video classification model to obtain the spraying video classification result; A fifth module is used to generate a target multimodal prompt according to the multimodal prompt template, the object model and the spraying video classification result; The sixth module is used to input the target multimodal prompt into the motion trajectory prediction model to obtain the robot motion trajectory; The seventh module is used to spray the workpiece according to the movement trajectory of the robot.

9. A computer device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images