Robot control method for tracking object surface and related equipment thereof

Abstract

The invention provides a robot control method for object surface tracing and related equipment thereof, and the method comprises the steps: a system receives a mode selection instruction of an administrator, and determines an execution scheme; image information of a to-be-operated target is acquired, a pre-trained artificial intelligence model is used to identify the number, position and category of the target in an image, and a 3D model of the target is reconstructed. And the system labels the operation area, generates a labeling box, and optimizes the 3D model. And by extracting and positioning feature points of the model and combining continuous change of the appearance, a motion track and a posture which can be executed by the robot are generated, and object surface tracing is completed. And finally, according to the execution scheme, the labeling box, the image information and the tracing result, the robot is controlled to perform operation on the target surface, efficient and accurate automatic operation is achieved, and potential safety hazards caused by manual object surface processing are avoided.

Description

Technical Field

[0001] The present invention relates to the field of robots, and in particular to a robot control method for tracking an object surface and related equipment. Background Art

[0002] Surface treatment of objects is one of the most common production processes in manufacturing, encompassing all aspects of the manufacturing industry. A significant portion of these processes involves generating trajectories and postures based on the surface characteristics and treatment process, driving the operation of surface treatment modules to remove, modify, or coat the target surface. This type of application is often used for mold cleaning, plasma surface treatment, and product spraying. Therefore, tire mold cleaning is particularly important, directly impacting tire production quality and requiring rapid and reliable cleaning. Traditional cleaning methods involve manual cleaning of the mold surface using spray / launch devices, including sandblasting, ultrasonic cleaning, carbon dioxide cleaning, and lasers. However, these methods generate toxic gases due to the vaporization of rubber, impacting the safety of the working environment and the health of workers. Summary of the Invention

[0003] In view of this, an embodiment of the present invention provides a robot control method for tracking an object surface and related equipment, aiming to solve the safety issues existing in the existing manual processing of object surfaces.

[0004] The present invention provides a robot control method for tracking an object surface, comprising: Receive mode selection instruction information from the administrator terminal and determine the execution plan according to the mode instruction information; Acquire image information of the target to be measured, where the image information of the target to be measured is image information of the target to be operated; Identify the image information to be tested based on the pre-trained artificial intelligence model, and obtain the number, location, and category of the targets to be operated in the image information to be tested; Reconstruct the 3D model of each target to be operated, mark the operating area in the target image according to the recognized target image information, and generate a marking box; Optimize the 3D model corresponding to each target to be operated according to the preset algorithm; Extract and locate the feature points of the 3D model, and obtain the continuous change of the shape of the target to be operated based on the characteristics and relative relationships of each feature point and the 3D model; According to the process requirements of the target to be worked on and the continuous changes in the shape of the target to be worked on, the robot generates a motion trajectory and execution posture that can be executed to complete the surface tracking of the object; According to the execution plan, the marking box, the image information of the target to be measured and the result of the object surface tracking, the robot is controlled to operate on the surface of each target to be operated.

[0005] Preferably, the operation area includes a safety area and an identification area. The steps of reconstructing a 3D model of each target to be operated, marking the operation area in the target image according to the identified target image information, and generating a marking frame include: Through synchronization and fusion of 2D and 3D images, the spatial coordinate system of the target to be operated is unified; By synchronizing image annotation with spatial data, the area where the target to be operated is located is divided into identification zones and safety zones; monitoring the status of a pallet, wherein the pallet is contained within the robot; Whether the target to be operated is completely within the identification area is determined by a preset method, wherein the preset method includes at least one of image filtering preprocessing, region of interest segmentation, mean operation, depth threshold limitation, and image area data comparison.

[0006] Preferably, the step of optimizing the 3D model corresponding to each target to be operated according to a preset algorithm includes: The 2D color image is fused with the 3D depth data, and the obtained camera internal and external parameters are calibrated using a coordinate system conversion algorithm to convert the depth camera's coordinate system to the color camera's coordinate system. The depth image is reprojected through the image reprojection algorithm to ensure that each pixel can be accurately mapped to the corresponding position of the RGB image; Through data alignment and synchronization, the region of interest is cropped from the RGB image and ensures a one-to-one correspondence between the region of interest and the corresponding pixels in the depth image; The point cloud is generated and the posture is calculated through the 3D model of the target to be operated, the selected depth image data is converted into a point cloud format, and the position and posture of the target to be operated in three-dimensional space are calculated.

[0007] Preferably, after performing point cloud generation and posture calculation using a 3D model of the target to be operated, converting the selected depth image data into a point cloud format, and calculating the position and posture of the target to be operated in three-dimensional space, the method further includes: Select the cutting axis along the X-axis, Y-axis, or Z-axis as required; Determine the cutting range and define the cutting limit based on the minimum and maximum values ​​in the specified axis; Automatically filter out qualified point cloud data according to the specified axis and cutting range; Points in the point cloud data are grouped into a plurality of clusters according to certain similarity criteria in a preset manner, wherein the preset manner includes at least one of a point cloud clustering algorithm, a spatial distance calculation algorithm, a cluster formation algorithm, and an object extraction algorithm.

[0008] Preferably, the step of extracting and locating the feature points of the 3D model, and obtaining a continuously changing shape of the target to be operated in combination with the 3D model based on the characteristics and relative relationships of each feature point, includes: Through the threshold definition procedure, a distance threshold is pre-set to determine whether the points belong to the same plane; Through random sampling and plane equation calculation, three points are randomly selected to construct the initial plane, and the distance from the point to the initial plane is calculated; Project all points onto the calculated plane and measure the actual distance from each point to the initial plane; Through the inlier determination and iterative optimization algorithm, the points whose distance from the screening point to the initial plane is lower than the preset threshold are set as inliers; Through continuous iterative optimization, until the plane with the maximum number of internal points is found; Through covariance matrix calculation and feature analysis, the covariance matrix of the point cloud data is calculated, and the eigenvalue decomposition of the covariance matrix is ​​performed to determine the main direction of the point cloud; Through the coordinate system transformation algorithm, a new coordinate system is constructed based on the eigenvectors to ensure that the new coordinate axes are consistent with the main directions of the point cloud; By calculating the boundary values ​​in the new coordinate system, the maximum and minimum values ​​of the point cloud along each axis are solved respectively; The minimum volume cube based on the boundary value is determined by bounding box calculation and inverse transformation algorithm, and converted back to the original coordinate system to complete the calculation of the minimum oriented bounding box, where the original coordinate system is the coordinate system of the depth camera.

[0009] Preferably, before the step of receiving the mode selection instruction information from the administrator terminal and determining the execution plan according to the mode instruction information, the following steps are included: Initialize the robot and its accessories through the robot's system-integrated driver to obtain the initial parameter file.

[0010] Preferably, the step of generating a point cloud and calculating a posture of the target to be operated by using a 3D model of the target to be operated, converting the selected depth image data into a point cloud format, and calculating the position and posture of the target to be operated in three-dimensional space includes: Divide the three-dimensional space into several small solids through the voxel grid; Through the point selection strategy, a representative point is selected from all the points in each small solid.

[0011] In a second aspect, the present invention further provides a robot control device for tracking an object surface, applying the above-mentioned robot control method for tracking an object surface, comprising: An interactive module is used to receive mode selection instruction information from an administrator terminal and determine an execution plan according to the mode instruction information; An image module is used to obtain image information of a target to be measured, where the image information of the target to be measured is image information of a target to be operated; The recognition module is used to identify the image information to be tested based on the pre-trained artificial intelligence model and obtain the number, location and category of the objects to be operated in the image information to be tested; The annotation module is used to reconstruct the 3D model of each target to be operated, annotate the operation area in the target image to be tested according to the recognized target image information, and generate an annotation box; Model optimization module, used to optimize the 3D model corresponding to each target to be operated according to the preset algorithm; The extraction module is used to extract and locate the feature points of the 3D model, and obtain the continuous change of the shape of the target to be operated based on the characteristics and relative relationships of each feature point and the 3D model; The tracking module is used to generate a motion trajectory and execution posture that can be executed by the robot according to the process requirements of the target to be operated and the continuous change of the target's appearance, thereby completing the tracking of the object surface; The execution module is used to control the robot to operate on the surface of each target to be operated according to the execution plan, the marking box, the image information of the target to be measured and the result of the object surface tracking.

[0012] In a third aspect, the present invention further provides an electronic device comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, wherein when the computer program instructions are executed by the processor, the method as described above is implemented.

[0013] In a fourth aspect, the present invention further provides a storage medium having computer program instructions stored thereon, wherein the method described above is implemented when the computer program instructions are executed by a processor.

[0014] In summary, the beneficial effects of the present invention are as follows: The robot control method for tracking the surface of an object provided in an embodiment of the present invention adopts 3D vision and 2D / 3D synchronization technology to realize the information collection of the appearance, shape, category, size, coordinates, and posture of the work object, obtains the point cloud in the object space based on the laser triangulation algorithm or the time-of-flight algorithm, and cuts and separates the point cloud through artificial intelligence visual recognition capabilities. Through algorithmic processing of the 3D point cloud, the actual size, shape, posture, and spatial coordinates of the object required by this application are finally obtained.

[0015] After that, the surface treatment planning calculation of the corresponding target is carried out and converted into the execution trajectory of the robot, thereby improving the robot's vision and spatial perception, and then facilitating the determination of the execution target, execution area and execution trajectory based on the robot's vision and spatial perception. Finally, the robot performs surface treatment operations according to the execution trajectory to achieve intelligent and highly flexible surface treatment effects that can adapt to products of different types and specifications, greatly increasing the scope of application of the surface treatment robot.

[0016] The information on the appearance, shape, category, size, coordinates, and posture of the work object is collected, and the point cloud in the object space is obtained based on the laser triangulation algorithm or time-of-flight algorithm. The point cloud is cut and separated through artificial intelligence visual recognition capabilities, and the actual size, shape, posture, and spatial coordinates of the object required by this application are finally obtained through algorithm processing of the 3D point cloud.

[0017] After that, the surface treatment planning calculation of the corresponding target is carried out and converted into the execution trajectory of the robot, thereby improving the robot's vision and spatial perception, and then facilitating the determination of the execution target, execution area and execution trajectory based on the robot's vision and spatial perception. Finally, the robot performs surface treatment operations according to the execution trajectory to achieve intelligent and highly flexible surface treatment effects that can adapt to products of different types and specifications, greatly increasing the scope of application of surface treatment robots and solving safety issues caused by manual processing of object surfaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work, and these are all within the scope of protection of the present invention.

[0019] Figure 1 The present invention is a flow chart of a first embodiment of a robot control method for tracking an object surface.

[0020] Figure 2 The present invention is a schematic structural diagram of a robot in one embodiment of a robot control method for tracking an object surface.

[0021] Figure 3 The present invention is a flowchart of the operation of a control system of a robot in one embodiment of a robot control method for tracking an object surface.

[0022] Figure 4 The present invention is a schematic diagram of an artificial intelligence model training process in one embodiment of a robot control method for tracking on the surface of an object.

[0023] Figure 5A schematic diagram of the application process of an artificial intelligence model in one embodiment of a robot control method for tracking on the surface of an object.

[0024] Figure 6 The present invention is a structural diagram of a robot control device for tracking on the surface of an object.

[0025] Figure 7 2 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In order to make the objects, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the specific embodiments described herein are only configured to explain the present invention and are not configured to limit the present invention. For those skilled in the art, the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present invention by illustrating examples of the present invention.

[0027] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, the elements defined by the phrase "comprising..." do not exclude the presence of other identical elements in the process, method, article, or device comprising the elements.

[0028] Reference Figures 1 to 4 The present invention provides a robot control method for tracking an object surface, comprising: S1: Receive mode selection instruction information from the administrator terminal and determine the execution plan according to the mode instruction information; S2: Acquire image information of the target to be measured, where the image information of the target to be measured is image information of the target to be operated; S3: Identify the image information to be tested based on the pre-trained artificial intelligence model, and obtain the number, location, and category of the targets to be operated in the image information to be tested; S4: reconstructing the 3D model of each target to be operated, marking the operating area in the target image according to the recognized target image information, and generating a marking frame; S5: Optimize the 3D model corresponding to each target to be operated according to the preset algorithm; S6: Extract and locate the feature points of the 3D model, and obtain the continuous change of the shape of the target to be operated based on the characteristics and relative relationships of each feature point and the 3D model; S7: Based on the process requirements of the target to be worked on and the continuous changes in the target's shape, a motion trajectory and execution posture that can be executed by the robot are generated to complete the surface tracking of the object; S8: According to the execution plan, the marking box, the image information of the target to be measured and the result of the object surface tracking, the robot is controlled to operate on the surface of each target to be operated.

[0029] In an embodiment of the present invention, the robot includes a robot body 1, a robot mounting fixture 2, a visual component 3, a surface treatment equipment terminal 4, an equipment machine 5 and a pallet 6, wherein the pallet 6 is used to process the working target, and the visual component 3 includes a 2D camera and a 3D camera.

[0030] Reference Figures 1 to 4 The robot includes a 2D / 3D camera, auxiliary light source, anti-fouling and cleaning structure, a multifunctional machine vision bracket, the robot body and integrated drive and control system, a standard robot workbench (including a base, control cabinet, debugging platform, robot lifting mechanism, and related equipment and accessories for the signal and system control center), an AI image recognition model, AI image annotation software, an AI image training program, an AI feature analysis model, a local server, a local database, process-based software and a human-machine interface, a server interaction system, and database project management software. The 2D vision camera, auxiliary light source, AI image annotation software, and AI image training program are used to generate the AI ​​recognition model. The 3D vision camera collects spatial information of the target to be processed through the robot's movement, enabling identification of the object's position and posture in space. The robot body, integrated drive and control system, and standard robot workbench serve as the device's action actuators. The local server, local database, process-based action execution software and human-machine interface, the server interaction system, and database project management software provide management, upgrades, service, and application support for the system.

[0031] Reference Figure 2 The robot's operating modes include manual mode, intelligent mode, and communication mode. In manual mode, the robot can execute a default solution or a manually selected solution. In intelligent mode, the robot can intelligently select a solution. In communication mode, the robot waits for instructions from the upper level to select a solution. At least one target to be operated is placed on the pallet 6. Vision component 3 acquires image information of the target to be operated. The image information to be operated is identified using a pre-trained artificial intelligence model, and the number, location, and category of the targets to be operated in the image information are determined.

[0032] The 3D model of each target to be worked on is reconstructed. Based on the recognized target image information, the work area in the target image is annotated and a labeling box is generated. Because the spatial coordinate system of the target to be worked on can deviate due to viewing angle deviation in the vision component 3, 2D / 3D image synchronization and fusion technology is used to optimize the 3D model of each target to be worked on. This optimized 3D model facilitates the subsequent extraction and location of 3D model feature points.

[0033] Extract and locate the feature points of the 3D model, and obtain the continuous change of the shape of the target to be processed based on the characteristics and relative relationships of each feature point and the 3D model. Specifically, according to the preset execution strategy, the guiding or associated structural features of the target to be processed are extracted separately, and then the edges, corners, and inner points of the extracted guiding features of the target to be processed are searched through edge detection algorithms (such as Canny edge detection), internal point detection algorithms, and corner point detection algorithms (such as Harris corner point detection). The continuous change of the shape of the target to be processed is obtained by combining the characteristics of each feature point and the relative relationships of edges, corners, and inner points with the reconstructed 3D model. Among them, the preset execution strategy is a pre-set plan, in which the guiding structural features of each target to be processed are stored. The robot obtains the guiding structural features based on the identified target to be processed.

[0034] Since different targets to be worked on have different process requirements, the process requirements and the continuous shape changes of the target to be worked on are combined to generate a motion trajectory and execution posture that can be executed by the robot to complete the surface tracking of the object. The specific tracking process is as follows: Through the path planning algorithm, for the round-trip path of the robotic arm and surface treatment operation, the equipment's motion range, obstacle distribution, speed and acceleration limits are comprehensively considered to find the shortest, least time-consuming and obstacle-avoiding safe path, recalculate the optimized path, and send the identified object coordinates, transition position, and offset parameters to the robot assembly to achieve reduced energy consumption and operation cycle. According to the execution plan, annotation box, image information of the target to be measured, and the results of object surface tracking, the robot is controlled to operate on the surface of each target to be worked on. The operation content includes but is not limited to cleaning, spraying, polishing, and surface hardening.

[0035] Specifically, by calling the camera driver provided by the 2D / 3D industrial camera manufacturer, integrating and optimizing it, we can achieve the adaptation, communication, control, data extraction, and data optimization of the 2D / 3D industrial camera. By using 2D vision and AI intelligent recognition, we can accurately identify the target to be operated in complex environments. 3D vision and 2D / 3D synchronization technology are used to collect information on the appearance, shape, category, size, coordinates, and posture of the work object. Based on the laser triangulation algorithm or time-of-flight algorithm, the point cloud in the space of the target to be worked is obtained, and the point cloud is cut and separated through artificial intelligence visual recognition capabilities. Through algorithmic processing of the 3D point cloud, the actual size, shape, posture, and spatial coordinates of the target to be worked required by this application are finally obtained.

[0036] After that, the surface treatment planning calculation of the corresponding target is carried out and converted into the execution trajectory of the robot, thereby improving the robot's vision and spatial perception, and then facilitating the determination of the execution target, execution area and execution trajectory based on the robot's vision and spatial perception. Finally, the robot performs surface treatment operations according to the execution trajectory to achieve intelligent and highly flexible surface treatment effects that can adapt to products of different types and specifications, greatly increasing the scope of application of the surface treatment robot.

[0037] In summary, simply replacing the functional equipment at the end of the robot can achieve different surface treatment scenarios. Compared to manual labor, robots significantly improve work efficiency, working hours, and workflow consistency. This saves time while avoiding mold damage, ensuring safer work, and saving raw materials. The resulting benefits are already directly enabling companies to achieve cost reduction and efficiency gains. A robot control method and system for surface tracking uses an artificial intelligence image model trained for targets of varying shapes and sizes. Through process-specific execution strategies, this method enables a robotic workstation control method for surface treatment to possess considerable intelligence and flexibility in practical application scenarios. This intelligent robot control method and system for surface tracking can quickly analyze, plan strategies, generate programs, and execute them for any target within the model range, achieving complete unmanned operation. After long-term model and strategy iteration, the target coverage and scope of this robot control method and system for surface tracking can be further improved, ultimately achieving intelligent processing of all target objects, self-iteration, and large-scale unmanned deployment, effectively addressing the safety concerns of personnel involved in surface treatment operations.

[0038] Furthermore, the operation area includes a safety area and an identification area, and the 3D model of each target to be operated is reconstructed. The operation area in the target image to be tested is annotated according to the identified target image information to generate the annotation frame step S2, including: S21: Unify the spatial coordinate system of the target to be operated by synchronizing and fusing 2D and 3D images; S22: By synchronizing image annotation with spatial data, the area where the target to be operated is divided into an identification zone and a safety zone; S23: monitoring the status of a pallet, wherein the pallet is contained in the robot; S24: Determine whether the target to be operated is completely within the identification area through a preset method, wherein the preset method includes at least one of image filtering preprocessing, region of interest segmentation, mean operation, depth threshold limitation, and image area data comparison.

[0039] In an embodiment of the present invention, by calling the camera driver provided by the manufacturer of the 2D / 3D industrial camera, integration and optimization are performed to realize the adaptation, communication, control, data extraction, and data optimization operations of the 2D / 3D industrial camera. Through 2D / 3D image synchronization and fusion technology, the technology of correcting the deviation of the spatial coordinate system of the target to be operated caused by the deflection of the viewing angle is used to unify the spatial coordinate system of the target to be operated, which is the spatial coordinate data of the 3D vision device. The purpose of image annotation is to mark the working area of ​​the target to be operated. This process can be done manually or by artificial intelligence software. Through the synchronization of image annotation and spatial data, the working position of the annotation box is automatically located according to the preset graphic corner recognition algorithm (2D image annotation tool, 2D / 3D point cloud mapping, 3D point cloud cutting, point cloud minimum circumscribed rectangle solution), and the identification area and safety area of ​​the area where the target to be operated are located are divided. The specific process for identifying corner points in an image is as follows: A 2D image annotation tool is used to obtain a labeling box. The pixels within the labeling box in the 2D photo are mapped to the same position in the 3D point cloud. The corresponding 3D point cloud within the labeling box is then cut out from the 3D point cloud. The smallest cuboid that completely encloses the 3D point cloud is calculated; this cuboid becomes the final labeling box. The recognition zone is the area where the vision component 3 identifies and measures the target to be worked on, and the safety zone is the area where the robot can move on the surface of the object. Laser sensors are used to identify and monitor the status of the pallet 6, which is the pallet on which the target to be worked on is placed. Image filtering preprocessing involves filtering the image to remove noise and improve image quality. Region of interest segmentation uses an object detection algorithm to extract the region of interest (ROI) of interest from the image. Meaning calculation calculates the average pixel value of a specified area of ​​the image (e.g., the ROI). This is typically used to evaluate image brightness, color, or other characteristics. Depth thresholding involves setting a threshold range based on depth information (e.g., the depth value in a 3D image) and retaining only points or pixels within this range. By comparing image area data, it is possible to determine whether a certain area in the image meets preset conditions, such as whether it is part of the target to be processed. In summary, the above method improves image quality to accurately determine whether the target to be processed is completely within the recognition area.

[0040] Furthermore, step S5 of optimizing the 3D model corresponding to each target to be operated according to a preset algorithm includes: S51: Fusing the 2D color image with the 3D depth data, calibrating the acquired camera internal and external parameters through a coordinate system conversion algorithm, and converting the depth camera's coordinate system to the color camera's coordinate system; S52: Reproject the depth image using an image reprojection algorithm to ensure that each pixel can be accurately mapped to the corresponding position of the RGB image; S53: cropping the region of interest from the RGB image through data alignment and synchronization, and ensuring that the region of interest corresponds to the corresponding pixel points in the depth image one-to-one; S54: generating a point cloud and calculating a posture of the target to be operated through the 3D model, converting the selected depth image data into a point cloud format, and calculating the position and posture of the target to be operated in the three-dimensional space.

[0041] In an embodiment of the present invention, the 2D color image is fused with the 3D depth data, and the internal and external parameters of the camera are calibrated and obtained through a coordinate system conversion algorithm, and the coordinate system of the depth camera is converted to the color camera coordinate system. The depth image is reprojected by an image reprojection algorithm to ensure that each pixel can be accurately mapped to the corresponding position of the RGB image. Through data alignment and synchronization, the area of ​​interest is cropped from the RGB image, and it is ensured that the area of ​​interest corresponds one-to-one with the corresponding pixel points in the depth image. The cropping of the image is completed by an artificial intelligence model. Point cloud generation and posture calculation are performed through the 3D model of the target to be operated, the selected depth image data is converted into a point cloud format, and the position and posture of the target to be operated in three-dimensional space are calculated.

[0042] Specifically, the image data source is provided by a 2D / 3D industrial camera. After the 2D / 3D industrial camera is integrated, calibrated, initialized, and debugged, a stable, high-quality 2D / 3D image that meets the requirements of this application is obtained.

[0043] Through point cloud generation algorithms and 2D / 3D image point cloud synchronization calibration algorithms, the images are pre-processed to obtain a color plane image of the target to be processed, construct a three-dimensional outline, and accurately measure the unit's geometric dimensions, providing data support for the next step of image processing. The AI ​​model (trained and optimized with a large number of image samples) and target detection obtained by AI-based process operations require rapid identification of different cargo categories and corresponding label locations. Therefore, this application uses a visual algorithm to fuse 2D color images with 3D depth data, achieving the effect of mapping the 2D image results cut by the AI ​​model to a 3D point cloud and processing them.

[0044] 3D vision and 2D / 3D synchronization technology are used to collect information on the appearance, shape, category, size, coordinates, and posture of the work object, and then perform surface treatment path planning calculations on the corresponding object and convert it into the robot's execution trajectory.

[0045] Through the depth map, RGB map, and synchronization information of the depth map and RGB map obtained by the 3D vision device, the spatial coordinate data of the 3D vision device coordinate system corresponding to the single or multiple target pixel points marked by the graphic annotation tool on the RGB map is read, and the coordinates are aggregated into the robot coordinate system through the machine vision annotation algorithm, and the execution order and compensation parameters of each coordinate are attached. The compensation parameters are analyzed through the operation process, combined with AI recognition, machine vision recognition, and the gap between the actual results of the robot execution and the process requirements is compensated. The optimal compensation parameters for this type of process are finally determined through multiple experiments and tests, and finally copied and executed as the compensation parameters of this type of process strategy. In this way, the purpose of specifying the robot coordinates through the visual field and forming the action execution logic is achieved.

[0046] Furthermore, after performing point cloud generation and posture calculation on the 3D model of the target to be operated, converting the selected depth image data into a point cloud format, and calculating the position and posture of the target to be operated in three-dimensional space in step S54, the following steps are included: S55: Select the cutting axis along the X-axis, Y-axis, or Z-axis as required; S56: Determine the cutting range based on the minimum and maximum values ​​in the specified axis and define the cutting limit; S57: Automatically filter out qualified point cloud data according to the specified axis and cutting range; S58: Grouping the points in the point cloud data into a plurality of clusters according to a certain similarity standard in a preset manner, wherein the preset manner includes at least one of a point cloud clustering algorithm, a spatial distance calculation algorithm, a cluster formation algorithm, and an object extraction algorithm.

[0047] In an embodiment of the present invention, point cloud data in three-dimensional space is composed of a large number of points, each with its own three-dimensional coordinates (x, y, z). To process this point cloud data, an axis (X, Y, or Z) can be selected to define a "cutting axis," and the point cloud can then be cut along this axis. The cutting range is determined based on the minimum and maximum values ​​along the specified axis, and the cutting boundaries are defined. Based on the specified axis and cutting range, point cloud data that meets the requirements is automatically filtered, and point cloud data outside the cutting range is discarded. Because an image may contain multiple objects to be processed, points in the point cloud data are grouped into multiple clusters based on certain similarity criteria using a preset method, with each cluster corresponding to a target to be processed. The preset method includes at least one of a point cloud clustering algorithm, a spatial distance calculation algorithm, a cluster formation algorithm, and an object extraction algorithm. The point cloud clustering algorithm groups points in a point cloud into multiple clusters based on certain similarity criteria. Each cluster contains a group of points with similar features. The spatial distance calculation algorithm calculates the distance between points in the point cloud to help determine the similarity between points. Clustering algorithms group points in a point cloud into clusters based on similarity criteria. This process typically involves initializing cluster centers, assigning points to clusters, and updating cluster centers. Object extraction algorithms extract individual objects from a point cloud. These algorithms typically build on the clustering results and further process each cluster to extract the object's features and boundaries.

[0048] Furthermore, step S6 of extracting and locating the characteristic points of the 3D model and obtaining a continuously changing shape of the target to be operated in combination with the 3D model according to the characteristics and relative relationships of each characteristic point includes: S61: presetting a distance threshold through a threshold definition program for determining whether the points belong to the same plane; S62: Randomly select three points to construct an initial plane through random sampling and plane equation calculation, and calculate the distance from the points to the initial plane; S63: Project all points onto the calculated plane and measure the actual distance between each point and the initial plane; S64: Using an interior point determination and iterative optimization algorithm, points whose distances from the points to the initial plane are less than a preset threshold are selected and set as interior points; S65: Optimize continuously through iteration until the plane with the maximum number of interior points is found; S66: Calculate the covariance matrix of the point cloud data through covariance matrix calculation and feature analysis, and perform eigenvalue decomposition on the covariance matrix to determine the main direction of the point cloud; S67: Use the coordinate system transformation algorithm to construct a new coordinate system based on the eigenvectors, ensuring that the new coordinate axes are consistent with the main directions of the point cloud; S68: Solve the maximum and minimum values ​​of the point cloud along each axis in the new coordinate system through the boundary value; S69: Determine a cube with the minimum volume based on the boundary value through bounding box calculation and inverse transformation algorithm, and convert it back to the original coordinate system to complete the calculation of the minimum oriented bounding box, wherein the original coordinate system is the coordinate system of the depth camera.

[0049] In an embodiment of the present invention, a threshold definition program is used to pre-set a distance threshold to determine whether points belong to the same plane. If the distance from a point to a plane is less than the threshold, the point is considered to be on the plane. Three points are randomly selected and used to confirm a plane. Then, the distances of all points to the plane are calculated. All points are projected onto the calculated plane, and the actual distances of each point to the initial plane are measured. Through the inlier determination and iterative optimization algorithm, points whose distances from the initial plane meet a preset threshold are screened and set as inliers. Through continuous iterative optimization, the plane with the maximum number of inliers is found. Through covariance matrix calculation and feature analysis, the covariance matrix of the point cloud data is calculated, and the eigenvalue decomposition is performed on the covariance matrix to determine the main direction of the point cloud. For example, the covariance matrix of the point cloud is calculated and the eigenvalue decomposition is performed to obtain three eigenvalues ​​and corresponding eigenvectors. Assume that the eigenvalues ​​are λ1> λ2> λ3, and the corresponding eigenvectors are v1, v2, and v3. Eigenvector v1 represents the longest extension direction (primary direction) of the point cloud, v2 represents the secondary longest direction, and v3 represents the shortest direction. Eigenvectors v1, v2, and v3 serve as the three axes of the new coordinate system. All points are then transformed from the original coordinate system to this new coordinate system. In the new coordinate system, the maximum and minimum values ​​of the point cloud data along each axis are calculated. This step is to determine the boundaries of the point cloud data. Based on the boundaries of the point cloud data, a cube with the minimum volume is determined that can completely contain the point cloud data. This cube is then transformed back to the original coordinate system, completing the calculation of the minimum oriented bounding box.

[0050] Reference Figure 3 , receiving the mode selection instruction information from the administrator terminal, and determining the execution plan according to the mode instruction information before step S1 includes: S0: Initialize the robot and its accessories through the robot's system integrated driver and obtain the initial parameter file.

[0051] In an embodiment of the present invention, the robot system integrates a robot and its associated equipment program to perform initial parameter tuning on the robot assembly. This includes initializing and monitoring the robot status, resetting the robot assembly to its initial position, and resetting all equipment to their initial states. Calibrate and synchronize the visual coordinate system of the 3D visual imaging device with the coordinates of the robot or its functional end device according to a preset machine vision calibration algorithm. Using the visual calibration data read by the intelligent robot system, the robot performs an initial positioning of the pattern blocks within its field of view to confirm the approximate location and number of the targets. A secondary fine positioning is then performed based on the target sequence. This positioning involves taking a close-up image of the target. Target classification, feature classification, and corner and inlier point retrieval are then performed according to preset algorithms (e.g., AI training and recognition, target geometric feature database, etc.). Exemplarily, this process involves obtaining the classification of the target to be processed through AI training and recognition, obtaining and classifying features on the target to be processed, and locating corner and inlier points of the target to be processed, thereby obtaining the complete features of the target to be processed. In other embodiments of the present invention, the geometric feature database can also be used to automatically obtain relevant feature information after identifying the specific type of the target to be processed.

[0052] The segmented area is calculated and the running trajectory is determined through filtering algorithms (the filtering algorithms include but are not limited to one or more of threshold segmentation, discrete filtering, mean filtering, contour recognition, minimum enclosing rectangle generation algorithm, point cloud scaling algorithm, point cloud difference cutting, point cloud ROI generation algorithm, point cloud mapping, plane fitting, surface intersection segmentation, intersection extraction, through filtering, point cloud threshold shrinkage, point cloud geometry correction, geometric distance solution, drop recognition, intersection positioning, threshold arrangement, point cloud threshold extraction, point cloud smoothing, point cloud segmentation fitting, single point associated point cloud area expansion, point set posture solution, and point set optimization sorting). The task running logic and the robot running trajectory and posture are determined according to the path.

[0053] Specifically, through the 2D / 3D industrial camera driver integrated in the robot system, the initial parameters of the 2D / 3D industrial camera are debugged and the status of the 2D / 3D industrial camera is monitored to generate an initial parameter file; through the robot and supporting equipment program integrated in the robot system, the initial parameters of the robot assembly are debugged and an initial parameter file is generated; the robot status is initialized and monitored, and the robot assembly is reset to the initial position; each device is reset to the initial state; through the hand-eye calibration software of the robot system and the robot hand-eye calibration program of the intelligent surface treatment robot, the 2D / 3D industrial camera and the robot assembly are calibrated on the eye-on-hand basis, and an executable calibration file is generated; illustratively, the visual calibration data read by the intelligent robot system is used to process the target within the field of view; according to the target's accuracy requirements for the operation, the system can choose whether to adjust the shooting position for the operation target again based on the results of the first positioning, and shoot the target again to obtain a clearer and more accurate image than the first shot, so as to perform a second precise positioning of the operation target in order to obtain an execution accuracy result that meets the process requirements of the operation target. This positioning process typically involves taking a close-up photo of the target. Pre-set algorithms are then used to perform target and feature classification, as well as corner and interior point identification, on a higher-quality point cloud. Filtering algorithms are then used to calculate segmented regions and determine the trajectory. The task logic and robot trajectory are then determined based on the path. The intelligent surface tracking robot system reads the AI ​​recognition model file required for the current solution, serving as the model foundation for subsequent recognition functions.

[0054] In practice, the coordinate system calibration software for the vision component 3 and the robot actuator primarily involves capturing images of the calibration object at various positions and postures using a camera, while simultaneously recording the position and posture of the robot arm's end effector in the same coordinate system. Feature points of the calibration object are extracted from the camera images, typically using corner detection or other feature extraction algorithms. The postures of the camera and robot arm are represented in axis-angle form, where the axis is a unit vector and the angle represents the magnitude of the rotation. Rotations can be converted to axis-angle form using the logarithmic projection of the rotation matrix. A linear system of equations is formed: The axis-angle posture data is organized into a linear system of equations in a specific format. This system can be expressed as AX = XB, where A and B are the axis-angle representations of the camera and robot arm, respectively, and X is the transformation matrix to be solved. The linear system of equations is solved using the least squares method, meaning that an optimal solution X is found such that AX ≈ XB. The optimal solution is obtained by converting the system of equations into matrix form and solving it using the least squares method. The transformation matrix between the camera and the robot arm is extracted from the optimal solution X, and finally unified data collection of the visual component 3 and the robot execution coordinate system is achieved.

[0055] Furthermore, step S54 of converting the selected depth image data into a point cloud format and calculating the position and posture of the target to be operated in three-dimensional space includes: S541: Divide the three-dimensional space into several small solids through voxel grid; S542: Select a representative point from all points in each small solid through a point selection strategy.

[0056] In an embodiment of the present invention, to improve the efficiency of image processing in this application while conserving computing resources on edge computing devices, a point cloud downsampling technique is employed to divide the three-dimensional space into a number of small cubes (voxels) through voxel gridding. A point selection strategy is employed to select a representative point for all points within each voxel using certain rules (such as average value, center point, etc.). After processing using the aforementioned downsampling algorithm, a significantly reduced point cloud dataset is output while maintaining the overall structure and features of the original point cloud. Ultimately, this simplifies the data volume without sacrificing key features, achieving the goals of improving efficiency and conserving edge computing power.

[0057] Reference Figure 4 and 5 The artificial intelligence of this application is achieved through artificial intelligence training algorithms and software, artificial intelligence annotation software (manual and intelligent annotation), artificial intelligence image training software, and artificial intelligence model data preprocessing and debugging software. By collecting a large number of 2D images of various situations of the application object of this application, and manually or intelligently screening, annotating, and combining them through this artificial intelligence annotation software, and performing 2D image training and parameter adjustment through this image training software and artificial intelligence model parameter debugging software, until a large artificial intelligence recognition model capable of realizing the artificial intelligence visual recognition function of this application is obtained; After obtaining a high-quality artificial intelligence model through the above-mentioned artificial intelligence-related software tools developed by this application, the trained instance segmentation model is converted into the ONNX format and loaded using the ONNX Runtime GPU module to realize model deployment; secondly, necessary preprocessing operations are performed on the input image, such as scaling and normalization, to adapt to the model requirements and complete the preprocessing of the model data; the preprocessed image is sent to the model, forward propagation is performed, and the prediction results are obtained to realize model inference; after the inference is completed, it is necessary to add a post-processing algorithm to post-process the data, filter low-quality prediction boxes by setting a confidence threshold, and apply the non-maximum suppression (NMS) algorithm to eliminate redundant detection boxes; the final output includes the bounding box, confidence score and classification label of each detected object, realize instance segmentation, and adapt to monomer recognition and segmentation in complex backgrounds.

[0058] In the artificial intelligence implementation process of this application, it is necessary to continuously optimize the above-mentioned artificial intelligence training algorithms and software, artificial intelligence labeling software (manual labeling and intelligent labeling), artificial intelligence image training software, artificial intelligence model data preprocessing and debugging software, and obtain advanced, innovative, efficient, accurate, stable, and flexible artificial intelligence model creation tools. After repeated efforts in object image collection, manual image screening, manual labeling, intelligent labeling, manual re-inspection, original labeled image screening, artificial intelligence training, model screening, model parameter adjustment, artificial intelligence model screening, artificial intelligence model merging, etc., practice, debugging, and testing, we will eventually obtain a high-quality intelligent model and ultimately obtain a large artificial intelligence model for a specific field with ultra-high recognition ability, ultra-high compatibility, ultra-high accuracy, ultra-high precision, and ultra-high efficiency.

[0059] Reference Figure 6 The present invention also provides a robot control device for tracking an object surface, and a robot control method for tracking an object surface, comprising: Interaction module 11, used to receive mode selection instruction information from the administrator terminal and determine the execution plan according to the mode instruction information; The image module 12 is used to obtain image information of the target to be measured, where the image information of the target to be measured is image information of the target to be operated; The recognition module 13 is used to recognize the image information to be tested based on the pre-trained artificial intelligence model, and obtain the number, location and category of the target to be operated in the image information to be tested; The annotation module 14 is used to reconstruct the 3D model of each target to be operated, annotate the operation area in the target image to be measured according to the identified target image information, and generate an annotation frame; Model optimization module 15, used to optimize the 3D model corresponding to each target to be operated according to a preset algorithm; An extraction module 16 is used to extract and locate the feature points of the 3D model, and obtain the continuous change of the shape of the target to be operated based on the characteristics and relative relationships of each feature point and the 3D model; The tracking module 17 is used to generate a motion trajectory and execution posture that can be executed by the robot according to the process requirements of the target to be operated and the continuous change of the target's appearance, so as to complete the tracking of the object surface; The execution module 18 is used to control the robot to operate on the surface of each target to be operated according to the execution plan, the marking box, the image information of the target to be measured and the result of the object surface tracking.

[0060] Reference Figures 1 to 4The robot includes a 2D / 3D camera, auxiliary light source, anti-fouling and cleaning structure, a multifunctional machine vision bracket, the robot body and integrated drive and control system, a standard robot workbench (including a base, control cabinet, debugging platform, robot lifting mechanism, and related equipment and accessories for the signal and system control center), an AI image recognition model, AI image annotation software, an AI image training program, an AI feature analysis model, a local server, a local database, process-based software and a human-machine interface, a server interaction system, and database project management software. The 2D vision camera, auxiliary light source, AI image annotation software, and AI image training program are used to generate the AI ​​recognition model. The 3D vision camera collects spatial information of the target to be processed through the robot's movement, enabling identification of the object's position and posture in space. The robot body, integrated drive and control system, and standard robot workbench serve as the device's action actuators. The local server, local database, process-based action execution software and human-machine interface, the server interaction system, and database project management software provide management, upgrades, service, and application support for the system.

[0061] Reference Figure 2The robot's working modes include manual mode, intelligent mode and communication mode. In manual mode, the robot can execute the default execution plan or the manually selected plan; in intelligent mode, the robot can intelligently select a plan; in communication mode, the robot waits for the upper-level instruction to select a plan. At least one target to be operated is placed on the pallet 6. The image information of the target to be operated is obtained through the visual component 3. The image information to be operated is identified according to the pre-trained artificial intelligence model, and the number, position and category of the targets to be operated in the image information to be operated are obtained. The 3D model of each target to be operated is reconstructed, and the operation area in the target image to be operated is marked according to the identified image information of the target to be operated, and a marking frame is generated. Since the visual component 3 will cause the spatial coordinate system of the target to be operated to deviate due to the deflection of the viewing angle, the 2D / 3D image synchronization and fusion technology is used to optimize the 3D model of each target to be operated. The optimized 3D model is more convenient for the subsequent extraction and positioning of the feature points of the 3D model. The feature points of the 3D model are extracted and positioned, and the continuous change of the shape of the target to be operated is obtained in combination with the 3D model based on the characteristics and relative relationships of each feature point. Specifically, according to a preset execution strategy, the guiding or associated structural features of the target to be processed are individually extracted. Edge, corner, and inliers of the extracted guiding features of the target to be processed are then searched for using edge detection algorithms (such as Canny edge detection), inlier detection algorithms, and corner detection algorithms (such as Harris corner detection). The continuous change in the shape of the target to be processed is obtained by combining the characteristics of each feature point and the relative relationships between edges, corners, and inliers with the reconstructed 3D model. The preset execution strategy is a pre-set plan that stores the guiding structural features of each target to be processed. The robot then obtains the guiding structural features based on the identified target to be processed.

[0062] Since different targets to be worked on have different process requirements, the process requirements and the continuous shape changes of the target to be worked on are combined to generate a motion trajectory and execution posture that can be executed by the robot to complete the surface tracking of the object. The specific tracking process is as follows: Through the path planning algorithm, for the round-trip path of the robotic arm and surface treatment operation, the equipment's motion range, obstacle distribution, speed and acceleration limits are comprehensively considered to find the shortest, least time-consuming and obstacle-avoiding safe path, recalculate the optimized path, and send the identified object coordinates, transition position, and offset parameters to the robot assembly to achieve reduced energy consumption and operation cycle. According to the execution plan, annotation box, image information of the target to be measured, and the results of object surface tracking, the robot is controlled to operate on the surface of each target to be worked on. The operation content includes but is not limited to cleaning, spraying, polishing, and surface hardening.

[0063] Specifically, by calling the camera driver provided by the 2D / 3D industrial camera manufacturer, integrating and optimizing it, it can achieve the adaptation, communication, control, data extraction, and data optimization of the 2D / 3D industrial camera. By using 2D vision and AI intelligent recognition, it can accurately identify the working object in complex environments. 3D vision and 2D / 3D synchronization technology are used to collect information on the appearance, shape, category, size, coordinates, and posture of the work object. The point cloud in the object space is obtained based on the laser triangulation algorithm or time-of-flight algorithm, and the point cloud is cut and separated through artificial intelligence visual recognition capabilities. Through algorithmic processing of the 3D point cloud, the actual size, shape, posture, and spatial coordinates of the object required by this application are finally obtained.

[0064] After that, the surface treatment planning calculation of the corresponding target is carried out and converted into the execution trajectory of the robot, thereby improving the robot's vision and spatial perception, and then facilitating the determination of the execution target, execution area and execution trajectory based on the robot's vision and spatial perception. Finally, the robot performs surface treatment operations according to the execution trajectory to achieve intelligent and highly flexible surface treatment effects that can adapt to products of different types and specifications, greatly increasing the scope of application of the surface treatment robot.

[0065] In summary, simply replacing the functional equipment at the end of the robot can achieve different surface treatment scenarios. Compared to manual labor, robots significantly improve work efficiency, working hours, and workflow consistency. This saves time while avoiding mold damage, ensuring safer work, and saving raw materials. The resulting benefits are already directly enabling companies to achieve cost reduction and efficiency gains. A robot control method and system for surface tracking uses an artificial intelligence image model trained for targets of varying shapes and sizes. Through process-specific execution strategies, this method enables a robotic workstation control method for surface treatment to possess considerable intelligence and flexibility in practical application scenarios. This intelligent robot control method and system for surface tracking can quickly analyze, plan strategies, generate programs, and execute them for any target within the model range, achieving complete unmanned operation. After long-term model and strategy iteration, the target coverage and scope of this robot control method and system for surface tracking can be further improved, ultimately achieving intelligent processing of all target objects, self-iteration, and large-scale unmanned deployment, effectively addressing the safety concerns of personnel involved in surface treatment operations.

[0066] In addition, combined Figure 7 The robot control method for object surface tracking according to the embodiment of the present invention may be implemented by an electronic device. Figure 7 A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention is shown.

[0067] The electronic device may include a processor 301 and a memory 302 storing computer program instructions.

[0068] Specifically, the processor 301 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured as one or more integrated circuits for implementing the embodiments of the present invention.

[0069] Memory 302 may include a large-capacity memory for data or instructions. By way of example, and not limitation, memory 302 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk, a magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 302 may include removable or non-removable (or fixed) media. Where appropriate, memory 302 may be internal or external to the data processing device. In certain embodiments, memory 302 is non-volatile solid-state memory. In certain embodiments, memory 302 includes read-only memory (ROM). Where appropriate, the ROM may be mask-programmable ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0070] The processor 301 reads and executes computer program instructions stored in the memory 302 to implement any one of the robot control methods for tracking on the surface of an object in the above embodiments.

[0071] In one example, the electronic device may further include a communication interface 303 and a bus 310. Figure 7 As shown, the processor 301 , the memory 302 , and the communication interface 303 are connected via a bus 310 and communicate with each other.

[0072] The communication interface 303 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiment of the present invention.

[0073] The bus 310 includes hardware, software, or both that couples components of the electronic device to each other. By way of example, and not limitation, the bus 310 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industrial Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, the bus 310 may include one or more buses 310. Although embodiments of the present invention describe and illustrate specific buses, the present invention contemplates any suitable bus 310 or interconnect.

[0074] In addition, in conjunction with the robot control method for tracking an object surface in the above-mentioned embodiments, embodiments of the present invention may provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when executed by a processor, the computer program instructions implement any of the robot control methods for tracking an object surface in the above-mentioned embodiments.

[0075] In summary, the embodiments of the present invention provide a robot control method for tracking an object surface and related equipment.

[0076] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0077] The functional blocks shown in the block diagrams described above can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they may be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, and the like. When implemented in software, the elements of the present invention are programs or code segments used to perform the desired tasks. Programs or code segments may be stored in a machine-readable medium or transmitted via a data signal carried in a carrier wave over a transmission medium or communication link. "Machine-readable medium" may include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memory, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and the like. Code segments may be downloaded via a computer network such as the Internet or an intranet.

[0078] It should also be noted that the exemplary embodiments described herein describe methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the steps described above. In other words, the steps may be performed in the order described in the embodiments, or in a different order, or several steps may be performed simultaneously.

[0079] The above description is only a specific embodiment of the present invention. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention.

Claims

1. A robot control method for tracking an object surface, characterized in that: include: receiving mode selection instruction information from an administrator terminal, and determining an execution plan according to the mode instruction information; Acquire image information of a target to be measured, wherein the image information of the target to be measured is image information of a target to be operated; Identify the image information to be tested according to a pre-trained artificial intelligence model, and obtain the number, location, and category of the target to be operated in the image information to be tested; Reconstructing a 3D model of each of the targets to be operated, marking the operating area in the target image according to the identified target image information, and generating a marking frame; Optimizing the 3D model corresponding to each of the targets to be operated according to a preset algorithm; Extracting and locating the characteristic points of the 3D model, and obtaining a continuous change of the shape of the target to be operated in combination with the 3D model according to the characteristics and relative relationships of each characteristic point; According to the process requirements of the target to be worked on and the continuous changes in the shape of the target to be worked on, a motion trajectory and execution posture that can be executed by the robot are generated to complete the surface tracking of the object; According to the execution scheme, the marking frame, the image information of the target to be measured and the result of the object surface tracking, the robot is controlled to operate on the surface of each target to be operated.

2. The robot control method for tracking an object surface according to claim 1, characterized in that: The operation area includes a safety area and an identification area. The steps of reconstructing the 3D model of each target to be operated, marking the operation area in the target image to be measured according to the identified image information of the target to be measured, and generating a marking frame include: Unify the spatial coordinate system of the target to be operated by synchronizing and fusing 2D and 3D images; By synchronizing image annotation with spatial data, the area where the target to be operated is located is divided into an identification zone and a safety zone; monitoring a status of a pallet, wherein the pallet is contained in the robot; Whether the target to be operated is completely within the identification area is determined by a preset method, wherein the preset method includes at least one of image filtering preprocessing, region of interest segmentation, mean operation, depth threshold limitation, and image area data comparison.

3. The robot control method for tracking an object surface according to claim 1, characterized in that: The step of optimizing the 3D model corresponding to each of the targets to be operated according to a preset algorithm includes: The 2D color image is fused with the 3D depth data, and the obtained camera internal and external parameters are calibrated using a coordinate system conversion algorithm to convert the depth camera's coordinate system to the color camera's coordinate system. The depth image is reprojected through the image reprojection algorithm to ensure that each pixel can be accurately mapped to the corresponding position of the RGB image; By aligning and synchronizing data, cropping a region of interest from the RGB image and ensuring a one-to-one correspondence between the region of interest and corresponding pixels in the depth image; Point cloud generation and posture calculation are performed using the 3D model of the target to be operated, the selected depth image data is converted into a point cloud format, and the position and posture of the target to be operated in three-dimensional space are calculated.

4. The robot control method for tracking an object surface according to claim 3, characterized in that: After the steps of generating a point cloud and calculating a posture of the target to be operated by using the 3D model, converting the selected depth image data into a point cloud format, and calculating the position and posture of the target to be operated in three-dimensional space, the method includes: Select the cutting axis along the X-axis, Y-axis, or Z-axis as required; Determine the cutting range and define the cutting limit based on the minimum and maximum values ​​in the specified axis; Automatically filter out qualified point cloud data according to the designated axis and the cutting range; Points in the point cloud data are grouped into a plurality of clusters according to a certain similarity standard in a preset manner, wherein the preset manner includes at least one of a point cloud clustering algorithm, a spatial distance calculation algorithm, a cluster formation algorithm, and an object extraction algorithm.

5. The robot control method for tracking an object surface according to claim 4, characterized in that: The step of extracting and locating the feature points of the 3D model, and obtaining a continuous change in the shape of the target to be operated based on the characteristics and relative relationships of each feature point and the 3D model, includes: Through the threshold definition procedure, a distance threshold is pre-set to determine whether the points belong to the same plane; By random sampling and plane equation calculation, three points are randomly selected to construct an initial plane, and the distance from the points to the initial plane is calculated; Project all points onto the calculated plane and measure the actual distance of each point to the initial plane; By using an interior point determination and iterative optimization algorithm, points whose distance from the screening point to the initial plane satisfies a preset threshold are set as interior points; Through continuous iterative optimization, until the plane with the maximum number of internal points is found; Calculating the covariance matrix of the point cloud data through covariance matrix calculation and feature analysis, and performing eigenvalue decomposition on the covariance matrix to determine the main direction of the point cloud; Through the coordinate system transformation algorithm, a new coordinate system is constructed based on the eigenvectors to ensure that the new coordinate axes are consistent with the main directions of the point cloud; By calculating the boundary values ​​in the new coordinate system, the maximum and minimum values ​​of the point cloud along each axis are solved respectively; The minimum volume cube based on the boundary value is determined by bounding box calculation and inverse transformation algorithm, and converted back to the original coordinate system to complete the calculation of the minimum oriented bounding box, wherein the original coordinate system is the coordinate system of the depth camera.

6. The robot control method for tracking an object surface according to claim 1, characterized in that: The step of receiving the mode selection instruction information from the administrator terminal and determining the execution plan according to the mode instruction information includes: The robot and its accessory devices are initialized through the system-integrated driver of the robot to obtain an initial parameter file.

7. The robot control method for tracking an object surface according to claim 3, characterized in that: The step of generating a point cloud and calculating a posture of the target to be operated by using the 3D model of the target to be operated, converting the selected depth image data into a point cloud format, and calculating the position and posture of the target to be operated in three-dimensional space includes: Divide the three-dimensional space into several small solids through the voxel grid; Through the point selection strategy, a representative point is selected from all the points in each small solid.

8. A robot control device for tracing on an object surface, applying the robot control method for tracing on an object surface according to any one of claims 1 to 7, characterized in that: include: An interaction module, configured to receive mode selection instruction information from an administrator terminal and determine an execution plan according to the mode instruction information; An image module is used to obtain image information of a target to be measured, wherein the image information of the target to be measured is image information of a target to be operated; An identification module is used to identify the image information to be tested based on a pre-trained artificial intelligence model, and obtain the number, location and category of the target to be operated in the image information to be tested; A labeling module is used to reconstruct the 3D model of each target to be operated, and to label the operating area in the target image according to the identified image information of the target to be measured, thereby generating a labeling frame; A model optimization module, configured to optimize the 3D model corresponding to each of the targets to be operated according to a preset algorithm; An extraction module is used to extract and locate the feature points of the 3D model, and obtain the continuous change of the shape of the target to be operated based on the characteristics and relative relationships of each feature point and the 3D model; A tracking module is used to generate a motion trajectory and execution posture that can be executed by the robot according to the process requirements of the object to be worked on and the continuous changes in the appearance of the object to be worked on, so as to complete the tracking of the object surface; An execution module is used to control the robot to operate on the surface of each target to be operated according to the execution scheme, the marking box, the image information of the target to be measured and the result of the object surface tracking.

9. An electronic device, characterized in that: include: At least one processor, at least one memory, and computer program instructions stored in the memory, which implement the method according to any one of claims 1 to 7 when the computer program instructions are executed by the processor.

10. A storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images