Intelligent spraying composite robot system based on large language model and method thereof

Abstract

The invention provides an intelligent spraying composite robot system based on a large language model and a method thereof, and relates to the field of intelligent robots and automatic spraying, the intelligent spraying composite robot system comprises a chassis, a mechanical arm, a spraying gun, a camera, an instruction receiver and a controller; a mechanical arm and an instruction receiver are carried on the chassis; spray guns and cameras are mounted on the mechanical arms; the instruction receiver receives a natural language voice instruction, analyzes semantics through a large language model, and generates an action instruction; the controller comprises a motion control module, a three-dimensional point cloud data acquisition module, a feature point extraction module, a track planning module and a spraying operation module; the method improves the man-machine interaction, reduces the use threshold of the robot, is easy to operate, gives play to the spraying advantages of the composite robot, does not need to program in mainstream offline simulation software, can generate a spraying path with adjustable and controllable parameters only by collecting the point cloud template of the workpiece and selecting the feature points, and can be widely applied to furniture, furniture and the like. And spraying operation in the fields of automobiles, aerospace and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent robots and automated spraying, and particularly relates to an intelligent spraying composite robot system and method based on a large language model. Background Art

[0002] At present, large language models have made remarkable progress in language understanding and reasoning, but they also face challenges in applying them to robot navigation and perception, such as how to enable robots to understand the environment and perform reliable path planning. Existing research mainly focuses on using large language models for language tasks or as decision-making agents, lacking an end-to-end framework that can input relevant natural language instructions from users and map information obtained by the robot's own perception, output a path to the target location, and complete corresponding perception tasks. Traditional motion planning algorithms, such as BFS and A* algorithms, have problems such as large computational amounts and non-smooth paths in complex environments. In addition, the challenge of virtual-real migration also limits the performance of robots in real environments.

[0003] Existing robot planning and perception technologies face problems such as insufficient integration of the semantic understanding and decision-making capabilities of large language models for robot navigation and perception tasks, limitations of traditional planning algorithms, challenges of virtual-real migration, and insufficient planning capabilities of large language models. Traditional path planning algorithms are inefficient in complex environments and generate non-smooth paths, and the differences between the simulation environment and the real environment also make it difficult for robots to apply the learned strategies to actual scenarios. In addition, how to make full use of the understanding and reasoning capabilities of large language models to generate better path planning schemes is also a challenge.

[0004] In summary, in the prior art, there are problems in intelligent spraying of robots, such as the spraying trajectory planning relying on manual teaching, the low efficiency of offline programming software, and the relatively high usage threshold lacking friendly human-computer interaction. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to provide an intelligent spraying composite robot system and method based on a large language model, which has high spraying efficiency and is easy to operate.

[0006] In a first aspect, the present invention provides an intelligent spraying composite robot system based on a large language model, including a chassis, a robotic arm, a spray gun, a camera, an instruction receiver, and a controller; The chassis is equipped with a robotic arm and an instruction receiver; the robotic arm is installed with a spray gun and a camera; The instruction receiver receives natural language voice instructions, parses the semantics through a large language model, and generates action instructions; The controller includes a motion control module, a three-dimensional point cloud data acquisition module, a feature point extraction module, a trajectory planning module, and a spraying operation module; A motion control module that, according to the action instruction, drives the chassis to move near the workpiece to be sprayed, and makes the camera located at the photographing position through the action of the robotic arm; A three-dimensional point cloud data acquisition module that controls the camera to take pictures of the workpiece and acquires the three-dimensional point cloud data of the workpiece; A feature point extraction module that extracts workpiece feature points through the three-dimensional point cloud data of the workpiece; A trajectory planning module that generates a spraying trajectory according to the workpiece feature points; A spraying operation module that controls the spray gun through the robotic arm to make the spray gun perform spraying operations according to the spraying trajectory.

[0007] Further, the feature point extraction module of the controller includes: performing voxel grid filtering, outlier filtering, and ROI (Region of Interest) region segmentation processing on the three-dimensional point cloud data, establishing multiple workpiece point cloud templates containing the coordinates of the spraying feature points, comparing the processed three-dimensional point cloud data with the multiple workpiece point cloud templates, and extracting the feature point coordinates of the matching workpiece point cloud template.

[0008] Further, the feature points of the workpiece point cloud template include: the four corners of the OBB (Opaque BinaryBlob) bounding box of the structured rectangular object, the cross-section corner points of the cylindrical object, and the highest point in the center line direction.

[0009] Further, the trajectory planning module generates a spraying trajectory according to the workpiece feature points, then calculates the robotic arm joint torque constraint according to the robotic arm dynamics model, and inserts transition points to generate a smooth trajectory.

[0010] Further, the process of the trajectory planning module generating a spraying trajectory includes: after receiving the feature point data, solving the maximum bounding rectangle of the feature points; calculating the normal vector of the plane where the feature points are located, and offsetting the maximum bounding rectangle along the normal vector direction of the plane by a preset spraying distance; calculating the trajectory inflection points according to the number of spray layers, and then connecting the inflection points in sequence.

[0011] Further, the spraying trajectory is a zigzag trajectory, and the generation steps include: S 1, when receiving four vertices P0, P2, P4, P6 in counterclockwise order as the feature points, using the pose differences P6 - P0 or P4 - P2 of the four feature points to find eight vectors AB, BA, BC, CB, CD, DC, DA, AD in the forward and reverse directions of the four sides; S 2, calculating the normal vector of the plane according to the plane where the feature points are located, and translating the feature points along the normal vector by the spraying distance; S 3, dividing the vector AD or BC in S 1 by the number of spray layers L to obtain the distance det(i) between each segment of the trajectory, where i is the segment number; S4. Calculate the coordinates of the intermediate points a_point(i) and b_point(i) from the teaching points P0 + i×AD÷L and P2 + i×BC÷L respectively. The coordinates of all intermediate points obtained by the recursive i method are used as the coordinates of the trajectory inflection points at both ends, and the coordinates of the trajectory inflection points at both ends are stored in the output list in the order of appearance.

[0012] Further, an IMU sensor is installed on the chassis. The inclination of the chassis is detected by the IMU sensor. If the inclination exceeds the threshold, the motion control module triggers an emergency stop.

[0013] Further, a collision detection module is installed on the robotic arm. The collision detection module starts reverse compensation based on current mutation and torque feedback.

[0014] Further, the action instruction includes a main body identifier, a target position / action code; the main body identifier is used to indicate the component that performs the action, and the components include the chassis, the robotic arm, and the camera. The target position represents the position where the chassis moves; the action code represents the operation type of the robotic arm.

[0015] On the other hand, the present invention provides an intelligent spraying method for an intelligent spraying composite robot system based on a large language model, including: Receive a natural language voice instruction, parse the semantics through the large language model, and generate an action instruction; According to the action instruction, the chassis moves to the vicinity of the workpiece to be sprayed, and the robotic arm moves to make the camera located at the photographing position; The camera takes a picture of the workpiece and collects the three-dimensional point cloud data of the workpiece; Extract the workpiece feature points according to the three-dimensional point cloud data of the workpiece; Generate a spraying trajectory according to the workpiece feature points; Control the spray gun through the robotic arm to make the spray gun perform spraying operations according to the spraying trajectory.

[0016] Beneficial effects: 1. Based on point cloud template matching and OBB bounding box feature point extraction, combined with the recursive Z-shaped trajectory algorithm, high-precision trajectory generation is achieved, and the efficiency is improved compared with traditional offline programming.

[0017] 2. Through the communication connection between the controller and the Mecanum wheel vehicle and the six-axis collaborative robotic arm of the composite robot based on Modbus_Tcp / IP, precise control of the actions of each component of the composite robot is achieved.

[0018] 3. Low-threshold human-computer interaction: The large language model and RAG technology directly map natural language instructions to action codes, and operators can control the composite robot without programming experience. Description of the drawings

[0019] In order to make the content of the present invention easier to be clearly understood, the present invention will be further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings.

[0020] Figure 1 This is the flowchart of Embodiment 2 of the present invention.

[0021] Figure 2 This is the process diagram of generating a smooth spraying trajectory in Embodiment 2 of the present invention. Detailed implementation manners

[0022] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments. The principles and features of the present invention will be described below with reference to the accompanying drawings. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The illustrated embodiments are only used to explain the present invention and are not intended to limit the scope of the present invention.

[0023] Embodiment 1 This embodiment discloses an intelligent spraying composite robot system based on a large language model to solve the problems in the prior art that spraying trajectory planning depends on manual teaching and offline programming software, resulting in low efficiency and a relatively high usage threshold for lack of friendly human-computer interaction.

[0024] This system includes a chassis, a robotic arm, a spray gun, a camera, an instruction receiver, and a controller; the chassis is equipped with a robotic arm and an instruction receiver; the robotic arm is installed with a spray gun and a camera; the instruction receiver receives natural language voice instructions, parses the semantics through a large language model, and generates action instructions; the controller includes a motion control module, a three-dimensional point cloud data acquisition module, a feature point extraction module, a trajectory planning module, and a spraying operation module; the motion control module drives the chassis to move near the workpiece to be sprayed according to the action instructions, and makes the camera located at the photographing position through the actions of the robotic arm; the three-dimensional point cloud data acquisition module controls the camera to take pictures of the workpiece and acquires the three-dimensional point cloud data of the workpiece; the feature point extraction module extracts workpiece feature points through the three-dimensional point cloud data of the workpiece; the trajectory planning module generates a spraying trajectory according to the workpiece feature points; the spraying operation module controls the spray gun through the robotic arm to make the spray gun perform spraying operations according to the spraying trajectory.

[0025] The feature point extraction module of the controller includes: performing voxel grid filtering, outlier filtering, and ROI region segmentation processing on the three-dimensional point cloud data, establishing multiple workpiece point cloud templates containing the coordinates of spraying feature points, comparing the processed three-dimensional point cloud data with the multiple workpiece point cloud templates, and extracting the feature point coordinates of the matching workpiece point cloud templates.

[0026] The feature points of the workpiece point cloud template include: the four corners of the OBB bounding box of a structured rectangular object, the cross-section corner points of a cylindrical object, and the highest point in the center line direction.

[0027] The trajectory planning module generates a spraying trajectory based on the workpiece feature points, then calculates the mechanical arm joint torque constraints according to the mechanical arm dynamics model, and inserts transition points to generate a smooth trajectory.

[0028] The process of the trajectory planning module generating a spraying trajectory includes: after receiving the workpiece feature point data, solving the maximum bounding rectangle of the workpiece feature points; calculating the normal vector of the plane where the workpiece feature points are located, and offsetting the maximum bounding rectangle along the direction of the plane normal vector by a preset spraying distance; calculating the trajectory inflection points according to the number of spraying layers, and then connecting the inflection points in sequence.

[0029] In this embodiment, a composite robot system including a Mecanum wheel vehicle chassis, a six-axis collaborative robotic arm, a Mech-Mind camera, and an industrial computer is constructed, and multi-node cooperative control is realized based on the ROS (Robot Operating System) framework. Through the Modbus_TCP / IP communication protocol, the target pose of the chassis navigation and the workpiece feature point coordinates are synchronously transmitted at the millisecond level, solving the problem of the disconnection between path planning and execution in the traditional scheme. For example, after the main control node parses the target position code, it synchronously triggers the AGV navigation service and the robotic arm pre-positioning service of the composite robot through the dynamic priority scheduling mechanism, enabling the chassis movement and the robotic arm preparation actions to be executed in parallel, and shortening the overall response time of the system.

[0030] A large language model using the Retrieval-Augmented Generation (RAG) technology constructs a semantic parsing layer to convert natural language instructions (such as "adjust the spraying height to 20 cm") into structured action codes. The action code format is defined as [execution entity identifier (0: AGV / 1: robotic arm / 2: camera) + target code + priority parameter]. For example, the instruction "emergency stop" is parsed into the action code "0xE001" and triggers the first-level safety protocol. Compared with the prior art, this solution realizes the dynamic binding of process parameters and action instructions, and users can directly adjust parameters such as spraying distance and flow rate through voice, improving the interaction flexibility.

[0031] The vision system collects three-dimensional point cloud data based on the Mech-Mind camera, and extracts workpiece feature points through voxel grid filtering, statistical outlier removal, and ROI region segmentation. For structured workpieces, a multi-template matching algorithm is used to calculate the four corner coordinates of the OBB bounding box, and the optimal pose of the spray gun is solved by combining principal component analysis. The spraying control module adopts a double-buffer mechanism: when receiving a trajectory generation signal, it completes the update of the trajectory file, and generates a Z-shaped trajectory through a recursive algorithm (for example, for a rectangular workpiece, based on the boundary vectors AB / AD, recursively calculate the intermediate point spacing Δ = spraying width / number of layers), ensuring that the trajectory generation delay is less than the specified time.

[0032] Set up a three-level interlock protection mechanism: 1. First-level protocol: The AGV chassis monitors the tilt angle in real time through the IMU, and triggers emergency braking when the limit is exceeded (response time is less than the specified time). 2. Second-level protocol: The robotic arm joints are equipped with a current mutation detection module. When the torque deviation exceeds the limit for 10 ms continuously, reverse compensation is started. 3. Third-level protocol: When spraying is abnormal, call the historical successful action code sequence, and select the nearest 3 compliance operations based on the timestamp for rollback recovery.

[0033] In a specific implementation, the trajectory planning is a dynamic trajectory generation method based on a multi-dimensional space coordinate system, which combines the geometric features of the workpiece and the spraying requirements, and can adjust the spraying path in real time to adapt to the shapes of different workpieces and spraying strategies. Preferably, the trajectory planning module adopts an adaptive algorithm, which can dynamically optimize the spraying path according to the workpiece point cloud data obtained in real time, so as to ensure the spraying uniformity and coverage rate.

[0034] By analyzing the point cloud data on the surface of the workpiece, multiple alternative trajectories are generated. Specifically, first use machine learning algorithms to classify the point cloud features of the workpiece, and identify different spraying areas and key feature points. Then, based on the identified feature points, multiple safe and efficient spraying trajectories can be generated, select the optimal path to avoid obstacles, and ensure the spraying quality.

[0035] The motion control module and the chassis navigation system are seamlessly connected through the Modbus_Tcp / IP protocol to ensure the movement and positioning accuracy during the spraying process. Through the real-time feedback mechanism, the chassis navigation system can accurately control the movement of the Mecanum wheel vehicle according to the path information generated by the trajectory planning module, ensuring that it always stays in the best position during the spraying process.

[0036] In terms of implementation details, the trajectory planning module can adopt a segmented control strategy, decomposing complex trajectories into multiple simple linear segments or curve segments. The parameters of each segment can be adjusted according to the spraying height, spraying speed and workpiece characteristics to ensure the best effect under different spraying conditions. At the same time, the module also introduces a collision detection algorithm to monitor the possible collision risks in the trajectory in real time, and trigger the safety mechanism in advance to ensure the safety of the equipment and operators.

[0037] The technical effects of the trajectory planning are reflected in: improving the flexibility and adaptability of spraying, shortening the spraying operation time, and significantly improving the spraying quality, avoiding spraying defects caused by improper path planning, and finally realizing efficient and accurate spraying operations.

[0038] ROS (Robot Operating System) is an open-source robot operating system that provides a series of tools and libraries for robot software development. ROS enables efficient information transfer and task collaboration between multiple nodes, supports distributed computing, allowing components in a composite robot system such as omnidirectional wheeled vehicles, robotic arms, and cameras to communicate in real time through ROS nodes.

[0039] ROS nodes include a spraying control node, a navigation control node, and a vision processing node, which exchange data with each other through topics and services. The spraying control node is responsible for receiving action codes from the master control node, parsing and generating corresponding control instructions, and sending the instructions to the robotic arm execution node through ROS topics to ensure precise control during the spraying process.

[0040] The navigation control node utilizes ROS path planning algorithms, in combination with the motion control system of the omnidirectional wheeled vehicle, to dynamically adjust the motion path based on real-time environmental data, ensuring that the omnidirectional wheeled vehicle can move safely and accurately to the target position. Preferably, the navigation control node can also be integrated with a lidar or a depth camera to obtain environmental information in real time, forming a complete environmental map to enhance the safety and flexibility of navigation.

[0041] The vision processing node captures the three-dimensional point cloud data of the workpiece through a Mech-Mind camera and performs feature point extraction and template matching through ROS image processing libraries. This node can send the extracted feature point data to the trajectory planning module through the ROS service interface to generate precise spraying trajectories. The vision processing node can also be combined with deep learning models to further improve the accuracy of feature point detection and matching.

[0042] The ROS system also supports a plug-in architecture, allowing users to expand functional modules according to their needs, such as adding new sensor nodes or control algorithms, facilitating future technology upgrades and system expansions. In addition, the ROS system can be connected to cloud services to achieve remote monitoring and analysis of data, promoting the intelligent and automated development of intelligent spraying systems.

[0043] Large language models can integrate Retrieval-Augmented Generation (RAG), enhancing the ability to understand natural language instructions through real-time connections with external knowledge bases or databases. It can not only extract relevant information from historical data but also dynamically retrieve and supplement data according to the context to ensure that the generated action codes have higher accuracy and adaptability.

[0044] RAG can be connected to a variety of data sources, such as real-time process databases, spray material characteristic databases, and workpiece historical spraying records. These data sources interact with the RAG module through standardized API interfaces, enabling it to dynamically obtain relevant background information when parsing natural language instructions and enhancing the intelligence level of instruction processing.

[0045] The RAG workflow is as follows: When receiving a natural language instruction, first parse the instruction content through a large language model. Subsequently, RAG retrieves relevant data based on the parsing result, extracts key information (such as spray material type, applicable process parameters, etc.), and then combines the retrieved information with the original instruction to generate a more accurate structured action code sequence. This sequence not only includes the execution subject identifier and target position but also parameter settings optimized for specific workpieces and spraying conditions.

[0046] After the camera collects the three-dimensional point cloud information of the workpiece, RAG can update the workpiece feature point database in real time to ensure that the latest workpiece features and spraying strategies are used in subsequent spraying tasks. In this way, the spraying path and parameters can be flexibly adjusted under different workpieces and spraying environments to achieve the best spraying effect.

[0047] RAG can play a greater role in multi-robot cooperation scenarios. For example, when parsing composite instructions containing spatial relationships (such as "Robot A moves to the work area and Robot B sprays"), it can identify and retrieve the task priorities and mutual relationships between robots to ensure the reasonable allocation and efficient execution of tasks. This not only improves the intelligence level of spraying operations but also enhances the flexibility and efficiency of multi-robot cooperation.

[0048] Voxel grid filtering is a point cloud data processing technology based on three-dimensional space discretization. Specifically, it divides the input point cloud data into uniform three-dimensional voxel grids to reduce the data volume and improve the subsequent processing efficiency. Preferably, the size of the voxel grid can be adaptively adjusted according to the size and characteristics of the target workpiece, so as to ensure that while retaining important details, redundant noise points are removed. By clustering the points within the voxel grid and calculating the geometric center of the points within each voxel or using the densest point as the representative point, a simplified point cloud is formed.

[0049] Voxel grid filtering can work in cooperation with other components such as the Mekanum camera and trajectory planning module. Specifically, the point cloud data processed by voxel grid filtering is sent to the feature point extraction algorithm to extract the key feature points of the workpiece. These feature points include the edges, corners, and other important geometric features of the workpiece, which will be used as the basis for subsequent spraying trajectory planning. By efficiently aggregating the points within the voxel grid, the speed and accuracy of feature point extraction can be significantly improved, thereby accelerating the response time of the entire spraying operation.

[0050] In addition, voxel grid filtering can also be combined with machine learning models. For example, deep learning algorithms can be used to classify and identify points within the voxel grid to meet the spraying requirements of different workpieces. By comparing the feature distributions of different workpieces after voxel grid filtering, the system can automatically select the optimal spraying parameters and trajectory planning strategies to achieve adaptive optimization of the spraying effect.

[0051] ROI region segmentation is a technical means in image processing. Its main function is to identify and extract the regions of interest in the image for subsequent feature extraction and analysis. Preferably, ROI region segmentation adopts a combination of multiple algorithms, such as the combination of convolutional neural network (CNN) based on deep learning and traditional image segmentation techniques, to improve the accuracy and efficiency of segmentation.

[0052] ROI region segmentation can perform real-time data transmission with a vision sensor (such as a Mekorma camera) through a high-speed data bus to ensure the rapid processing and feedback of image data. At the same time, the segmented ROI data can be directly transmitted to the controller through the Modbus_Tcp / IP protocol to dynamically adjust the spraying trajectory and parameters according to the extracted feature points.

[0053] Furthermore, ROI region segmentation can also be combined with a three-dimensional point cloud data acquisition module. By analyzing the geometric features of the point cloud data, accurate identification of the target region in three-dimensional space can be achieved. This method can not only improve the spraying coverage rate but also effectively reduce waste during the spraying process and improve the material utilization rate.

[0054] In addition, ROI region segmentation can also be linked with a machine learning model. Historical data is used to train the model to achieve adaptive identification and segmentation of the characteristics of different workpieces. This adaptive ability enables the robot to quickly adjust the segmentation strategy when facing workpieces of different shapes and materials, thus ensuring the consistency and high quality of the spraying effect.

[0055] In terms of technical effects, the system combined with ROI region segmentation can improve the intelligent level of spraying operations while efficiently processing image data, enabling the composite robot to more flexibly handle complex production environments and diverse workpiece shapes, and further optimizing the spraying path planning and execution efficiency.

[0056] The OBB bounding box is an oriented bounding box, specifically the optimal bounding box determined by performing principal component analysis (PCA) on the point cloud data of the workpiece. Preferably, the boundaries of the bounding box are consistent with the main shape features of the workpiece to achieve precise enclosure of the workpiece contour. The construction of the OBB bounding box not only considers the geometric shape of the workpiece but also the orientation of the workpiece in space, enabling more precise control of the spraying angle and spraying area during the spraying process.

[0057] Preferably, each boundary of the OBB bounding box can be linked with the spraying head of the robotic arm. Through a real-time feedback mechanism, the posture of the spraying head is adjusted to ensure that the spraying coverage range is consistent with the boundary of the OBB bounding box. In addition, the corner points of the OBB bounding box can serve as key points for the movement trajectory of the robotic arm. By collecting and analyzing these key points, a more refined spraying path can be generated, reducing the possible phenomena of missed spraying or over-spraying during the spraying process.

[0058] It can receive point cloud data update information in real time to dynamically adjust the parameters of the bounding box. When the shape of the workpiece changes, it can automatically recalculate the OBB bounding box and send the updated boundary information to the controller via the Modbus_Tcp / IP protocol to timely adjust the spraying strategy.

[0059] Furthermore, the boundaries of the OBB bounding box can be combined with the spraying height parameters in the spraying control module to form a three-dimensional spraying area. This area not only considers the surface features of the workpiece but also combines the spraying range and spraying distance of the spraying equipment, making the liquid distribution during the spraying process more uniform. By setting different spraying heights within the OBB bounding box, it is possible to adapt to different workpiece surface features and improve the spraying quality and efficiency.

[0060] To enhance the intelligence level of the system, the OBB bounding box can be linked with the semantic action conversion module of the large language model. When receiving a new spraying instruction, it can automatically adjust the parameters of the OBB bounding box according to the instruction content to ensure the rationality and effectiveness of the spraying path, thereby improving the flexibility and adaptability of the overall spraying operation.

[0061] The spraying operation module is an intelligent control system based on multi-level data processing and real-time feedback. Through an integrated multi-sensor system, it monitors the changes in the spraying environment in real time, including temperature and humidity sensors, air pressure sensors, and viscosity sensors of the spraying material, etc., to dynamically adjust the spraying parameters. Preferably, the sensors communicate with the controller via a wireless network to ensure real-time data transmission and processing, thereby improving the consistency of the spraying effect and the spraying quality.

[0062] The spraying operation module can include linkage with the vision system, obtaining real-time image data of the spraying area through the Mech-Mind camera, analyzing the state of the spraying surface in combination with image processing algorithms, so as to automatically adjust parameters such as spraying height, spraying speed, and spraying pressure during the spraying process to ensure spraying uniformity. Preferably, the vision system can identify defects on the workpiece surface and feedback them to the spraying control module to achieve intelligent compensation during the spraying process.

[0063] The spraying operation module can also share data with an industrial computer or a PC, perform machine learning using historical spraying data, and optimize the spraying strategy. By analyzing the success and failure cases of previous spraying tasks, the controller can automatically adjust the spraying parameters and path planning, forming an adaptive learning mechanism to improve spraying efficiency and accuracy.

[0064] This embodiment can achieve efficient information interaction between ROS nodes, accurately send the key coordinate poses to the trajectory planning module based on the Modbus_Tcp / IP communication method, and send the target position to the motion control module to improve the coordination and efficiency of spraying operations and chassis navigation.

[0065] Through the large language model, natural language instructions (such as "Go to the second workbench for spraying") are mapped to action codes containing the executor identifier and target position encoding. The instruction receiver parses the instructions and controls the corresponding robotic arm or Mecanum wheel vehicle through the controller.

[0066] Based on the Mech-Mind camera, the processing program performs voxel grid filtering, outlier filtering, and ROI region segmentation on the point cloud data, performs multi-template matching according to the point cloud template of the workpiece. The template contains the key feature point coordinates (such as the four corners of the OBB bounding box of a structured rectangular object, the cross-section corner points of a cylindrical object, and the highest point in the z direction). Solve the spray gun pose corresponding to the feature points according to the main components of the object's point cloud, and send the flag bits of the required trajectory and the key feature poses of the workpiece to the trajectory planning module.

[0067] Receive and cache parameters such as plane type, trajectory type, spraying height, pre-spraying amount, key point information, and trajectory file generation signal through modbus communication; when receiving the trajectory file generation signal, read the file directory and determine whether there is a previously generated file (if so, delete it, if not, continue to execute) to prevent file redundancy in the case of high generation frequency; calculate the sorting of all key points of the trajectory by the trajectory algorithm and store them in an array, and generate the trajectory tip and tid files in the sorted order. The time from receiving the parameters to generating the file does not exceed the specified time, and a generation completed signal is returned; when receiving the trajectory execution signal, start the remote file execution program, determine whether the robot is under external control, if so, execute the program, if not, return an error. The program executes the currently generated trajectory file, and after execution, returns a completion signal.

[0068] Compared with the traditional solution, this system improves the human-machine interaction, reduces the threshold for using robots, is easy to operate, gives full play to the advantages of composite robot spraying, and does not require programming in mainstream offline simulation software. Only by collecting the point cloud template of the workpiece and selecting feature points can a spray path with adjustable parameters be generated, which can be widely applied to spray painting operations in fields such as furniture, automobiles, and aerospace.

[0069] Example 2 As Figure 1 shown, this embodiment discloses an intelligent spraying method for an intelligent spraying composite robot system based on a large language model, including the following steps: S11, Receive a natural language voice command, parse the semantics through the large language model, and generate an action command; S12, According to the action command, the chassis moves to near the workpiece to be sprayed, and the camera is positioned at the photographing position through the action of the robotic arm; S13, The camera takes a picture of the workpiece to collect the three-dimensional point cloud data of the workpiece; S14, Extract the workpiece feature points according to the three-dimensional point cloud data of the workpiece; S15, Generate a spraying trajectory according to the workpiece feature points; S16, Control the spray gun through the robotic arm to make the spray gun perform spraying operations according to the spraying trajectory.

[0070] Specifically, the spraying steps of this embodiment include: S21, First, say the wake-up word "Xiaoyan" to the wireless microphone carried by the robot. The robot replies "I'm here" in voice, and then say a natural language control command, such as "Go to Workbench No. 1 and spray the workpiece on it"; S22, Then, the large model of the spraying host computer parses the command, converts it into an instruction array in the format of [instruction code, instruction content, instruction code, instruction content...], and sends it to the main control node; S23, The main control node sequentially traverses the instructions and calls the corresponding services according to the instruction array. Send the position information of Workbench No. 1 to the navigation program encapsulated in the AGV through modbus_tcp, and the AGV autonomously navigates to Workbench No. 1; S24, After the AGV arrives at the workbench, return the service end flag, and the main control node continues to call the camera photographing service and the robotic arm spraying service; S25, The end of the robotic arm moves to a fixed photographing pose, and the camera collects the three-dimensional point cloud information of the workpiece; S26, The program in the industrial computer or computer processes the three-dimensional point cloud, performs algorithms such as filtering and template matching to extract feature points, and sends them to the robotic arm control unit; S27. The trajectory planning module generates a robotic arm trajectory execution file based on the received feature point data, and the controller executes the trajectory file to achieve the movement of the robotic arm.

[0071] The controller of this embodiment inputs the feature point data into the structured workpiece full-path coverage planning algorithm to generate a smooth spraying trajectory. The controller calculates the joint torque constraint based on the robotic arm dynamics model to avoid trajectory overshoot. As Figure 2 shown, it specifically includes the following steps: S31. Taking the z-shaped trajectory as an example, set the received feature points as P0, P2, P4, P6. Use the 4 vertex coordinates to subtract each other to obtain 8 vectors AB, BA, BC, CB, CD, DC, DA, AD in the positive and negative directions of the 4 sides; obtain the pose differences P6 - P0, P4 - P2 of the 4 vertices. S32. Calculate the plane where the feature points are located, calculate the plane normal vector, and translate the feature points along the normal vector by the spraying distance d, keeping the pose unchanged. S33. Divide the vectors in S31 by the number of spraying layers AD / L, BC / L to obtain the spacing det(i) of each trajectory segment; at the same time, divide the pose by the number of spraying layers, so that the intermediate points in S34 contain uniform pose information (P6 - P0) / L. S34. Calculate the coordinates of the intermediate points a_point(i) and b_point(i) respectively from the teaching points P0 + i×AD÷L, P2 + i×BC÷L. All the intermediate points obtained by the recursive i method are sequentially input into the created Output_list list (a_point0, b_point0, a_point1, b_point1, a_point2, b_point2, a_point3, b_point3, a_point_4, b_piont4); S35. Sort the intermediate points in the Output_list list and connect them in order with the robotic arm linear execution command to generate a robotic arm execution file. S36. Control the movement of the robotic arm according to the robotic arm execution file.

[0072] The system of this embodiment sets up a multi-level security protocol: First-level protocol: Detect the chassis inclination through the IMU sensor of the Mecanum wheel vehicle to trigger the emergency stop threshold. Second-level protocol: The robotic arm collision detection module starts reverse compensation based on current mutation and torque feedback. Third-level protocol: When spraying is abnormal, call the historical successful action code sequence for recovery.

[0073] The method of this embodiment can be extended to multi-robot cooperation scenarios. By parsing composite instructions containing spatial relationships (such as "Xiaoyan moves to area A, and collaborative robot B processes component C") through a large language model, cross-device action codes are generated and task priorities are assigned.

[0074] Obviously, the above embodiments are merely examples for clear illustration and not limitations on the implementation manners. For those of ordinary skill in the art, other different forms of changes or variations can be made based on the above description. It is not necessary and impossible to enumerate all implementation manners here. And the obvious changes or variations derived therefrom still fall within the protection scope of the present invention.

Claims

1. An intelligent spraying composite robot system based on a large language model, characterized in that, It includes a chassis, a robotic arm, a spray gun, a camera, an instruction receiver, and a controller; The robotic arm and the instruction receiver are mounted on the chassis; The spray gun and the camera are installed on the robotic arm; The instruction receiver receives natural language voice instructions, parses the semantics through a large language model, and generates action instructions; The controller includes a motion control module, a three-dimensional point cloud data acquisition module, a feature point extraction module, a trajectory planning module, and a spraying operation module; The motion control module, according to the action instructions, drives the chassis to move near the workpiece to be sprayed, and makes the camera located at the photographing position through the actions of the robotic arm; The three-dimensional point cloud data acquisition module controls the camera to take pictures of the workpiece and acquires the three-dimensional point cloud data of the workpiece; The feature point extraction module extracts workpiece feature points through the three-dimensional point cloud data of the workpiece; The trajectory planning module generates a spraying trajectory according to the workpiece feature points; The spraying operation module controls the spray gun through the robotic arm to make the spray gun perform spraying operations according to the spraying trajectory.

2. The system according to claim 1, characterized in that The feature point extraction module of the controller includes: performing voxel grid filtering, outlier filtering, and ROI region segmentation processing on the three-dimensional point cloud data, establishing multiple workpiece point cloud templates containing the coordinates of spraying feature points, comparing the processed three-dimensional point cloud data with the multiple workpiece point cloud templates, and extracting the feature point coordinates of the matching workpiece point cloud templates.

3. The system according to claim 2, wherein The feature points of the workpiece point cloud template include: the four corners of the OBB bounding box of the structured rectangular object, the cross-section corner points of the cylindrical object, and the highest point in the center line direction.

4. The system according to claim 2, wherein The trajectory planning module generates a spraying trajectory according to the workpiece feature points, then calculates the joint torque constraints of the robotic arm according to the robotic arm dynamics model, and inserts transition points to generate a smooth trajectory.

5. The system according to claim 4, wherein The process of the trajectory planning module generating a spraying trajectory includes: after receiving the workpiece feature point data, solving the maximum bounding rectangle of the workpiece feature points; calculating the normal vector of the plane where the workpiece feature points are located, and offsetting the maximum bounding rectangle along the normal vector direction by a preset spraying distance; calculating the trajectory inflection points according to the number of spray layers, and then connecting the inflection points in sequence.

6. The system according to claim 5, characterized in that The spraying trajectory is a zigzag trajectory, and the generation steps include: S1, when the workpiece feature points received are the four vertices P0, P2, P4, P6 in counterclockwise order, using the attitude differences P6 - P0 or P4 - P2 of the four workpiece feature points, calculate eight vectors AB, BA, BC, CB, CD, DC, DA, AD in the positive and negative directions of the four sides; S2, calculate the plane normal vector according to the plane where the workpiece feature points are located, and translate the workpiece feature points along the normal vector by the spraying distance; S3, divide the vector AD or BC in S1 by the number of spray layers L to obtain the spacing det(i) of each section of the trajectory, where i is the section number; S4, calculate the coordinates of the intermediate points a_point(i) and b_point(i) respectively from the teaching points P0 + i × AD ÷ L and P2 + i × BC ÷ L, and use the coordinates of all intermediate points obtained by the recursive i method as the coordinates of the trajectory inflection points at both ends, and store the coordinates of the trajectory inflection points at both ends in the output list in sequence.

7. The system according to claim 1, characterized in that, The chassis is installed with an IMU sensor, and the chassis tilt is detected through the IMU sensor. When the tilt exceeds the threshold, the motion control module triggers an emergency stop.

8. The system according to claim 1, wherein The robotic arm is equipped with a collision detection module, which starts reverse compensation based on sudden current changes and torque feedback.

9. The system according to claim 1, wherein The action instructions include a main body identifier and a target position / action code; the main body identifier is used to indicate the component that performs the action, and the components include the chassis, the robotic arm, and the camera. The target position represents the position where the chassis moves. The action code represents the operation type of the robotic arm.

10. An intelligent spraying method using the system according to any one of claims 1-9, characterized in that, It includes: Receiving natural language voice instructions, parsing the semantics through a large language model, and generating action instructions. According to the action instructions, the chassis moves to near the workpiece to be sprayed, and the camera is positioned at the photographing position through the action of the robotic arm. The camera takes pictures of the workpiece and collects the three-dimensional point cloud data of the workpiece. According to the three-dimensional point cloud data of the workpiece, the feature points of the workpiece are extracted. According to the feature points of the workpiece, a spraying trajectory is generated. The spray gun is controlled by the robotic arm to perform spraying operations according to the spraying trajectory.

Images