3D vision active guidance trajectory generation method, device, equipment and medium for multi-material products

Abstract

The embodiment of the present invention discloses a 3D visual active guidance trajectory generation method, device, equipment and medium for multi-material products, wherein the method includes: scanning the product to be processed to obtain a point cloud to be processed; capturing feature points on the point cloud to be processed to obtain a feature point set to be processed, and extracting the region of each material to be analyzed from the point cloud to be processed to obtain a single material region to be processed; determining each contour point detection frame to be processed according to the feature point set to be processed, the template feature point set and the anchoring relationship between the template feature point and the detection frame; extracting a single material contour point set to be processed for each single material region to be processed according to each contour point detection frame to be processed; adjusting the template guidance trajectory according to each single material contour point set to be processed and each template single material contour point set to obtain a guide trajectory to be processed corresponding to the product to be processed. Thus, the guidance trajectory of the multi-material product is generated, and the effect of automated dispensing or spraying operation is improved.

Description

Technical Field

[0001] The present invention relates to the field of automated processing technology, and in particular to a method, device, equipment and medium for generating a 3D vision active guidance trajectory for a multi-material product. Background Art

[0002] Multi-material products are made of a combination of multiple materials. Traditional dispensing machines or spraying machines can no longer meet the needs, and there is currently no visual guidance track for multi-material products. Take the palm rest of an ergonomic keyboard as an example. The palm rest is made of multiple materials. The outermost layer is cloth, and the innermost layer is plastic. There is a certain amount of cloth left. The remaining cloth is needed to wrap the plastic. Brush adhesive on the remaining cloth and a certain area of ​​plastic. Use cloth to wrap the plastic. Because the palm rest has a certain amount of deformation and the length of the remaining cloth varies, it is generally only possible to apply glue manually. At present, there is no good solution for automated dispensing or spraying operations. Therefore, there is an urgent need for a method to generate guidance tracks for multi-material products. Summary of the invention

[0003] Based on this, it is necessary to address the above-mentioned technical problem of being unable to generate guidance trajectories for multi-material products, and propose a 3D vision active guidance trajectory generation method, device, equipment and medium for multi-material products.

[0004] This application proposes a 3D vision active guidance trajectory generation method for multi-material products, the method comprising:

[0005] Scan the product to be processed to obtain a point cloud to be processed;

[0006] Capturing feature points of the point cloud to be processed to obtain a feature point set to be processed, and extracting regions of each material to be analyzed from the point cloud to obtain a single material region to be processed;

[0007] Acquire a template feature point set of a template product, an anchoring relationship between the template feature point and a detection frame, a single-material contour point set of each template, and a template guide trajectory, wherein the template product and the product to be processed are the same type of product;

[0008] Determine the detection frame of each contour point to be processed according to the feature point set to be processed, the template feature point set and the anchoring relationship between the template feature point and the detection frame;

[0009] Extracting a set of single-material contour points to be processed for each single-material area to be processed according to each of the contour point detection frames to be processed;

[0010] According to each of the to-be-processed single-material contour point sets and each of the template single-material contour point sets, the template guide trajectory is adjusted to obtain the to-be-processed guide trajectory corresponding to the to-be-processed product.

[0011] Furthermore, the step of capturing feature points of the point cloud to be processed to obtain a set of feature points to be processed includes:

[0012] Inputting the to-be-processed point cloud into a target detection model corresponding to the template product to detect fixed features, and obtaining each fixed feature detection frame;

[0013] Searching for a center point within a range corresponding to each of the fixed feature detection frames in the point cloud to be processed as a single feature center point;

[0014] Each of the single feature center points is used as the feature point set to be processed.

[0015] Furthermore, the step of extracting the region of each material to be analyzed from the point cloud to be processed to obtain the single material region to be processed includes:

[0016] Using a preset filter combination method, filtering the point cloud to be processed to obtain a point cloud to be extracted, wherein the filter combination method is a method obtained by combining at least one of mean filtering, median filtering, Gaussian filtering and cluster filtering;

[0017] The point cloud to be extracted is input into a region segmentation model corresponding to the template product to perform region segmentation corresponding to each material to be analyzed, so as to obtain the single material region to be processed.

[0018] Furthermore, the step of extracting a set of single-material contour points to be processed for each single-material area to be processed according to each of the contour point detection frames to be processed includes:

[0019] Searching for boundary extreme value points within a range corresponding to each of the to-be-processed contour point detection boxes corresponding to the first material in the to-be-processed single material area corresponding to the first material, to obtain to-be-processed single material contour points, wherein the first material is any one of the various to-be-analyzed materials;

[0020] Each of the to-be-processed single-material contour points corresponding to the first material is used as the to-be-processed single-material contour point set corresponding to the first material.

[0021] Furthermore, before the step of obtaining the template feature point set of the template product, the anchoring relationship between the template feature point and the detection frame, the single material contour point set of each template and the template guide trajectory, it also includes:

[0022] Scanning the template product to obtain a template point cloud;

[0023] Capturing feature points of the template point cloud to obtain the template feature point set, and extracting regions of each material to be analyzed from the template point cloud to obtain a template single material region;

[0024] Acquire each template contour point detection frame input by the user according to each template single material area;

[0025] According to each of the template contour point detection frames corresponding to the second material, extracting the template single-material contour point set for the template single-material region corresponding to the second material, wherein the second material is any one of the various materials to be analyzed;

[0026] Determining the anchoring relationship between the template feature points and the detection frame according to each of the template contour point detection frames and the template feature point set;

[0027] Teaching the single material trajectory of the mechanism corresponding to each material to be analyzed on the template product;

[0028] The template guide trajectory is determined according to the single material trajectory of each mechanism.

[0029] Furthermore, the step of determining the anchoring relationship between the template feature points and the detection frame according to each of the template contour point detection frames and the template feature point set includes:

[0030] Acquire any one of the template contour point detection frames from each of the template contour point detection frames as a detection frame to be anchored;

[0031] Calculate the shortest distance between each feature point in the template feature point set and the detection frame to be anchored to obtain the shortest distance of a single point;

[0032] Find the shortest distance at the single point with the smallest value from among the shortest distances at the single point, and use it as the target distance;

[0033] Taking the point in the template feature point set corresponding to the target distance as the anchor point corresponding to the detection frame to be anchored;

[0034] The spatial position relationship between the anchor point corresponding to the detection frame to be anchored and the detection frame to be anchored is used as the single frame anchor relationship corresponding to the detection frame to be anchored;

[0035] Repeat the step of acquiring any one of the template contour point detection frames as the detection frame to be anchored, until the acquisition of the template contour point detection frames in the template contour point detection frames is completed;

[0036] Each of the single frame anchoring relationships is used as the anchoring relationship between the template feature point and the detection frame.

[0037] Furthermore, the step of determining the template guide trajectory according to each of the single material trajectories of the mechanism includes:

[0038] Obtaining the conversion relationship between the mechanism coordinates and the image coordinates corresponding to the template product;

[0039] According to the conversion relationship, each of the mechanism single material trajectories is mapped to the image coordinate system to obtain the image single material trajectory;

[0040] Based on the shortest path principle, the single-material trajectories of each image are spliced ​​to obtain the template guiding trajectory.

[0041] The present application also proposes a 3D vision active guidance trajectory generation device for a multi-material product, the device comprising:

[0042] Point cloud acquisition module, used to scan the product to be processed and obtain the point cloud to be processed;

[0043] A point cloud analysis module is used to capture feature points of the point cloud to be processed to obtain a feature point set to be processed, and to extract regions of each material to be analyzed from the point cloud to be processed to obtain a single material region to be processed;

[0044] A data acquisition module, used to acquire a template feature point set of a template product, an anchoring relationship between the template feature point and the detection frame, a single-material contour point set of each template, and a template guide trajectory, wherein the template product and the product to be processed are the same type of product;

[0045] A module for determining a detection frame of contour points to be processed, used to determine the detection frame of each contour point to be processed according to the set of feature points to be processed, the set of template feature points and the anchoring relationship between the template feature points and the detection frame;

[0046] A module for determining a single-material contour point set to be processed, configured to extract a single-material contour point set to be processed for each single-material region to be processed according to each of the single-material contour point detection frames to be processed;

[0047] The module for determining the guide trajectory to be processed is used to adjust the template guide trajectory according to each of the single-material contour point sets to be processed and each of the single-material contour point sets of the template to obtain the guide trajectory to be processed corresponding to the product to be processed.

[0048] The present application also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the following steps:

[0049] Scan the product to be processed to obtain a point cloud to be processed;

[0050] Capturing feature points of the point cloud to be processed to obtain a feature point set to be processed, and extracting regions of each material to be analyzed from the point cloud to obtain a single material region to be processed;

[0051] Acquire a template feature point set of a template product, an anchoring relationship between the template feature point and a detection frame, a single-material contour point set of each template, and a template guide trajectory, wherein the template product and the product to be processed are the same type of product;

[0052] Determine the detection frame of each contour point to be processed according to the feature point set to be processed, the template feature point set and the anchoring relationship between the template feature point and the detection frame;

[0053] Extracting a set of single-material contour points to be processed for each single-material area to be processed according to each of the contour point detection frames to be processed;

[0054] According to each of the to-be-processed single-material contour point sets and each of the template single-material contour point sets, the template guide trajectory is adjusted to obtain the to-be-processed guide trajectory corresponding to the to-be-processed product.

[0055] The present application also proposes a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor performs the following steps:

[0056] Scan the product to be processed to obtain a point cloud to be processed;

[0057] Capturing feature points of the point cloud to be processed to obtain a feature point set to be processed, and extracting regions of each material to be analyzed from the point cloud to obtain a single material region to be processed;

[0058] Acquire a template feature point set of a template product, an anchoring relationship between the template feature point and a detection frame, a single-material contour point set of each template, and a template guide trajectory, wherein the template product and the product to be processed are the same type of product;

[0059] Determine the detection frame of each contour point to be processed according to the feature point set to be processed, the template feature point set and the anchoring relationship between the template feature point and the detection frame;

[0060] Extracting a set of single-material contour points to be processed for each single-material area to be processed according to each of the contour point detection frames to be processed;

[0061] According to each of the to-be-processed single-material contour point sets and each of the template single-material contour point sets, the template guide trajectory is adjusted to obtain the to-be-processed guide trajectory corresponding to the to-be-processed product.

[0062] The 3D vision active guidance trajectory generation method for multi-material products of the present application first analyzes the feature point set to be processed and the single material area to be processed from the point cloud to be processed of the product to be processed, and then determines each contour point detection frame to be processed of the product to be processed based on the correspondence between the feature point set to be processed and the template feature point set and the anchoring relationship between the template feature point and the detection frame, and then extracts the single material contour point set to be processed for each single material area to be processed according to each contour point detection frame to be processed, and finally adjusts the template guidance trajectory according to each single material contour point set to be processed and each template single material contour point set to determine the guidance trajectory to be processed suitable for the product to be processed, thereby generating a guidance trajectory for the multi-material product, providing a basis for automated dispensing or spraying operations for the multi-material product, and improving production efficiency; and adjusting the template guidance trajectory according to each single material contour point set to be processed and each template single material contour point set, fully considering the deformation of the product to be processed, improving the accuracy of the determined guidance trajectory to be processed, and improving the effect of automated dispensing or spraying operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0064] in:

[0065] Figure 1 is a flow chart of a method for generating a 3D vision active guidance trajectory for a multi-material product in one embodiment;

[0066] Figure 2 It is a structural block diagram of a 3D vision active guidance trajectory generation device for multi-material products in one embodiment;

[0067] Figure 3 FIG. 4 is a structural block diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] like Figure 1 As shown, in one embodiment, a method for generating a 3D visual active guidance trajectory for a multi-material product is provided. The method can be applied to both a terminal and a server, and this embodiment is described by taking application to a terminal as an example.

[0070] The 3D vision active guidance trajectory generation method for multi-material products specifically includes the following steps:

[0071] S1: Scan the product to be processed to obtain the point cloud to be processed;

[0072] Specifically, a preset robot arm is used to grab the product to be processed and move it to the visual tool. The visual tool scans the product to be processed and uses the scanned point cloud as the point cloud to be processed.

[0073] The point data set of the product appearance surface obtained by visual tools in reverse engineering is also called point cloud.

[0074] Optionally, the visual tool of the present application adopts a three-dimensional line scan camera.

[0075] S2: capturing feature points of the point cloud to be processed to obtain a feature point set to be processed, and extracting regions of each material to be analyzed from the point cloud to obtain a single material region to be processed;

[0076] The feature points of the point cloud to be processed are representative points of fixed features that are not easily deformed in the product to be processed. For example, in the palm rest of an ergonomic keyboard, there is a fixed buckle, and there are no other parts near the buckle, so the buckle is used as a fixed feature.

[0077] Specifically, representative points among the points corresponding to the fixed features are extracted from the point cloud to be processed as feature points, and the extracted feature points are used as a feature point set to be processed.

[0078] The selection range of representative points includes: the center point, the point at the lower left corner, the point at the upper left corner, the point at the upper right corner, and the point at the lower right corner.

[0079] The material to be analyzed is the material you want to analyze. The value range of the material includes but is not limited to: plastic and cloth.

[0080] Specifically, the region of each material to be analyzed is extracted from the point cloud to be processed, and each extracted region is used as a single material region to be processed. That is, the single material region to be processed includes multiple points, and each point in the single material region to be processed represents the same material.

[0081] S3: Acquire a template feature point set of a template product, an anchoring relationship between the template feature point and the detection frame, a single material contour point set of each template, and a template guide trajectory, wherein the template product and the product to be processed are the same type of product;

[0082] Specifically, the template feature point set of the template product, the anchoring relationship between the template feature points and the detection frame, the single material contour point set of each template and the template guide trajectory can be obtained from the database, and the template feature point set of the template product, the anchoring relationship between the template feature points and the detection frame, the single material contour point set of each template and the template guide trajectory can also be obtained from the third-party application.

[0083] Since the template product and the product to be processed are the same product, except that the product to be processed may be deformed (also called deformed) relative to the template product, the goal of this application is to accurately determine the guide trajectory even when the product to be processed is deformed.

[0084] The template feature point set includes at least one feature point, wherein the feature point is a representative point of a fixed feature in the template product that is not easily deformed.

[0085] The anchoring relationship between the template feature points and the detection frame is the spatial relative position relationship between the feature points of the template product and the template contour point detection frame of the template product.

[0086] Optionally, the anchoring relationship between the template feature point and the detection frame is the spatial relative position relationship between the feature point of the template product and the center point of the template contour point detection frame of the template product.

[0087] The template single material contour point set is a collection of points of the contour of a material to be analyzed in the template product.

[0088] The template guide track is the visual guide track of the template product.

[0089] Optionally, the template guide trajectory is a guide trajectory in an image coordinate system of a vision tool.

[0090] S4: determining detection frames of contour points to be processed according to the feature point set to be processed, the template feature point set, and the anchoring relationship between the template feature points and the detection frame;

[0091] Specifically, firstly, the feature points in the feature point set to be processed are matched with the feature points in the template feature point set, and the two corresponding feature points are taken as a feature point pair; secondly, the spatial relative position relationship between the two feature points in the feature point pair is determined; then, according to the anchoring relationship between the template feature point and the detection frame, the template contour point detection frame corresponding to the first feature point of each feature point pair (that is, the feature point belonging to the template feature point set) is determined; finally, according to the spatial relative position relationship of the feature point pair, the template contour point detection frame corresponding to the first feature point of the feature point pair is transformed to obtain the contour point detection frame to be processed corresponding to the second feature point of the feature point pair (that is, the feature point belonging to the feature point set to be processed).

[0092] It can be understood that the two feature points in a feature point pair correspond to the same fixed feature.

[0093] S5: extracting a set of single-material contour points to be processed for each single-material area to be processed according to each of the contour point detection frames to be processed;

[0094] Specifically, a preset contour point determination method is adopted to search for contour points among the points in the single material area to be processed that correspond to the contour point detection box to be processed, and the searched contour points are used as the single material contour points to be processed; and the single material contour points to be processed corresponding to the same single material area to be processed are used as a single material contour point set to be processed.

[0095] The points in the to-be-processed single-material contour point set are points on the contour of the to-be-processed single-material region.

[0096] S6: adjusting the template guide trajectory according to each of the to-be-processed single-material contour point sets and each of the template single-material contour point sets to obtain the to-be-processed guide trajectory corresponding to the to-be-processed product.

[0097] Specifically, the spatial relative position relationship of each material to be analyzed is first determined based on each of the single-material contour point sets to be processed and each of the single-material contour point sets of the template. Then, based on the spatial relative position relationship of the material to be analyzed, the trajectory segment corresponding to the material to be analyzed in the template guide trajectory is adjusted. After the adjustment of the trajectory segments corresponding to all materials to be analyzed of the template guide trajectory is completed, the template guide trajectory is used as the guide trajectory to be processed corresponding to the product to be processed.

[0098] It can be understood that after the step of adjusting the template guide trajectory according to each of the single-material contour point sets to be processed and each of the single-material contour point sets of the template to obtain the guide trajectory to be processed corresponding to the product to be processed, it also includes: according to the conversion relationship between the mechanism coordinates and the image coordinates corresponding to the template product, mapping the coordinates of the guide trajectory to be processed corresponding to the product to be processed from the image coordinate system of the visual tool to the mechanism coordinates of the manipulator, and taking the mapped trajectory as the target guide trajectory; and performing gluing or spraying operations on the product to be processed according to the guide trajectory.

[0099] The conversion relationship between the mechanism coordinates and the image coordinates corresponding to the template product is a pre-calibrated conversion relationship. It is understandable that the implementation method of the conversion relationship between the mechanism coordinates and the image coordinates corresponding to the template product can be determined by those skilled in the art without creativity from the prior art, and will not be elaborated here.

[0100] This embodiment first analyzes the feature point set to be processed and the single material area to be processed from the point cloud to be processed of the product to be processed, and then determines each contour point detection frame to be processed of the product to be processed based on the correspondence between the feature point set to be processed and the template feature point set and the anchoring relationship between the template feature point and the detection frame, and then extracts the single material contour point set to be processed for each single material area to be processed according to each contour point detection frame to be processed, and finally adjusts the template guide trajectory according to each single material contour point set to be processed and each template single material contour point set to determine the guide trajectory to be processed suitable for the product to be processed, thereby generating a guide trajectory for a multi-material product, providing a basis for automated dispensing or spraying operations for multi-material products, and improving production efficiency; and adjusting the template guide trajectory according to each single material contour point set to be processed and each template single material contour point set, fully considering the deformation of the product to be processed, improving the accuracy of the determined guide trajectory to be processed, and improving the effect of automated dispensing or spraying operations.

[0101] In one embodiment, the step of capturing feature points of the point cloud to be processed to obtain a feature point set to be processed includes:

[0102] S211: Inputting the point cloud to be processed into a target detection model corresponding to the template product to detect fixed features, and obtaining each fixed feature detection frame;

[0103] Specifically, the point cloud to be processed is input into the target detection model corresponding to the template product to perform fixed feature detection, and the detection frame of each fixed feature obtained by detection is used as a fixed feature detection frame.

[0104] The target detection model is used to detect each point corresponding to the fixed features in the point cloud to be processed, and to generate a rectangular detection frame for each point corresponding to each fixed feature. The target detection model is a model obtained by training the target detection network. It is understandable that the specific training method of the target detection model can be determined by those skilled in the art without creativity from the prior art, and will not be elaborated here.

[0105] S212: searching for a center point within a range corresponding to each of the fixed feature detection frames in the point cloud to be processed as a single feature center point;

[0106] Specifically, a center point is searched out from each point in the to-be-processed point cloud within a range corresponding to each of the fixed feature detection frames, and the searched center point is used as a single feature center point.

[0107] That is to say, the fixed feature detection frame corresponds one-to-one to the single feature center point.

[0108] S213: Taking each of the single feature center points as the feature point set to be processed.

[0109] Specifically, each of the single feature center points is used as the feature point set to be processed, thereby determining a set of representative points of fixed features that are not easily deformed in the product to be processed.

[0110] This embodiment performs fixed feature detection through a target detection model, thereby realizing automated fixed feature detection and improving the degree of automation of this application; through the accurate detection performance of the target detection model, the accuracy of the determined feature point set to be processed is improved.

[0111] In one embodiment, the step of extracting the region of each material to be analyzed from the point cloud to be processed to obtain the single material region to be processed includes:

[0112] S221: Using a preset filter combination method to filter the point cloud to be processed to obtain a point cloud to be extracted, wherein the filter combination method is a method obtained by combining at least one of mean filtering, median filtering, Gaussian filtering and cluster filtering;

[0113] Specifically, a preset filtering combination method is used to filter the stray points in the point cloud to be processed, and the point cloud to be processed that has completed the filtering process is used as the point cloud to be extracted.

[0114] Image filtering, that is, suppressing the noise of the image corresponding to the point cloud to be extracted while retaining the image detail features corresponding to the point cloud to be extracted as much as possible, is an indispensable operation in image preprocessing. The quality of its processing effect will directly affect the effectiveness and reliability of subsequent image processing and analysis.

[0115] Wherein, when the filtering combination method is a method obtained by combining two or more filtering methods, each filtering method is executed in a preset order.

[0116] Optionally, when the filtering combination method is a method obtained by combining two or more filtering methods, if the filtering combination method includes clustering filtering, clustering filtering is used as the first method to be executed in the filtering combination method.

[0117] S222: Input the point cloud to be extracted into the region segmentation model corresponding to the template product to perform region segmentation corresponding to each material to be analyzed, so as to obtain the single material region to be processed.

[0118] Specifically, the point cloud to be extracted is input into a regional segmentation model corresponding to the template product to perform regional segmentation corresponding to each material to be analyzed, and the region segmented for each material to be analyzed is used as a single material region to be processed.

[0119] The region segmentation model is used to detect the points corresponding to each material to be analyzed in the point cloud to be extracted, and the points corresponding to each material to be analyzed are regarded as a region. The region segmentation model is a model obtained by training the target detection network. It is understandable that the specific training method of the region segmentation model can be determined by those skilled in the art without creativity from the prior art, and will not be elaborated here.

[0120] This embodiment uses a regional segmentation model to perform regional segmentation corresponding to each material to be analyzed, thereby realizing automated regional segmentation and improving the degree of automation of the present application; and firstly adopts a preset filtering combination method to filter the point cloud to be processed, thereby removing stray points, which is beneficial to improving the accuracy of subsequent regional segmentation.

[0121] In one embodiment, the step of extracting a set of single-material contour points to be processed for each single-material region to be processed according to each of the detection frames of contour points to be processed comprises:

[0122] S51: searching for boundary extreme value points within a range corresponding to each of the to-be-processed contour point detection boxes corresponding to the first material in the to-be-processed single material area corresponding to the first material, to obtain to-be-processed single material contour points, wherein the first material is any one of the various to-be-analyzed materials;

[0123] Each of the single material regions to be processed corresponds to a direction attribute. The value range of the direction attribute includes: maximum in the x direction, maximum in the y direction, maximum in the z direction, minimum in the x direction, minimum in the y direction, and minimum in the z direction. Through the direction attribute, the contour points of the single material to be processed can be accurately and quickly determined.

[0124] Specifically, among the points within the range corresponding to each of the contour point detection boxes to be processed corresponding to the first material in the single material area to be processed corresponding to the first material, boundary extreme value points with the same directional attributes as the contour point detection box to be processed are searched, and the searched boundary extreme value points are used as the single material contour points to be processed.

[0125] For example, if the direction attribute of the to-be-processed contour point detection box A1 corresponding to the first material is the largest in the x direction, then the largest point in the x direction is searched for among the points in the range corresponding to each to-be-processed contour point detection box corresponding to the first material in the to-be-processed single material area corresponding to the first material, and the searched point is used as the to-be-processed single material contour point corresponding to the to-be-processed contour point detection box A1.

[0126] S52: taking each of the to-be-processed single-material contour points corresponding to the first material as the to-be-processed single-material contour point set corresponding to the first material.

[0127] Specifically, each of the to-be-processed single-material contour points corresponding to the first material is taken as the to-be-processed single-material contour point set corresponding to the first material, that is, the points in the to-be-processed single-material contour point set all represent the first material.

[0128] It can be understood that by repeatedly executing step S51 to step S52, the to-be-processed single-material contour point set corresponding to each of the to-be-analyzed materials can be determined.

[0129] This embodiment uses the search boundary extreme point to quickly determine the to-be-processed single-material contour point set corresponding to each of the to-be-analyzed materials, thereby providing a basis for determining the to-be-processed guide trajectory corresponding to the to-be-processed product based on the to-be-processed single-material contour point set.

[0130] In one embodiment, before the step of obtaining the template feature point set of the template product, the anchoring relationship between the template feature point and the detection frame, the single material contour point set of each template, and the template guide trajectory, the following step is also included:

[0131] S31: Scan the template product to obtain a template point cloud;

[0132] Specifically, a preset manipulator is used to grab the template product and move it under the visual tool. The visual tool scans the template product and uses the scanned point cloud as the template point cloud.

[0133] S32: capturing feature points of the template point cloud to obtain the template feature point set, and extracting the region of each material to be analyzed from the template point cloud to obtain a template single material region;

[0134] Specifically, representative points among the points corresponding to the fixed features are extracted from the template point cloud as feature points, and the extracted feature points are used as the template feature point set.

[0135] Specifically, the template point cloud is subjected to region extraction for each material to be analyzed, and each extracted region is used as a template single material region. That is, the template single material region includes multiple points, and each point in the template single material region corresponds to the same material.

[0136] S33: acquiring each template contour point detection frame input by the user according to each template single material area;

[0137] Specifically, each template single-material area is displayed, and each template contour point detection frame input by the user is obtained according to the displayed template single-material area.

[0138] The template contour point detection frame is a rectangular detection frame. In other words, the template contour point detection frame is a three-dimensional frame.

[0139] Each of the template contour point detection boxes corresponds to a direction attribute.

[0140] S34: extracting the template single-material contour point set for the template single-material region corresponding to the second material according to each of the template contour point detection frames corresponding to the second material, wherein the second material is any one of the various materials to be analyzed;

[0141] Specifically, from each point in the corresponding range corresponding to the second material in the template single material area corresponding to the second material, search for the direction attribute of the template contour point detection box corresponding to the second material, use the searched boundary extreme point as the template single material contour point, and use each template single material contour point corresponding to the second material as the template single material contour point set corresponding to the second material.

[0142] It can be understood that by repeatedly executing step S34, the template single-material contour point set corresponding to each material to be analyzed can be determined.

[0143] S35: determining the anchoring relationship between the template feature points and the detection frame according to each of the template contour point detection frames and the template feature point set;

[0144] Specifically, the feature point in the template feature point set that is closest to the template contour point detection frame is used as the anchor point, and the spatial relative position relationship between the anchor point and the template contour point detection frame is used as the single frame anchor relationship; all single frame anchor relationships are used as the anchor relationship between the template feature point and the detection frame.

[0145] S36: Teaching the single material trajectory of the mechanism corresponding to each material to be analyzed on the template product;

[0146] Specifically, the single material trajectory of the mechanism corresponding to each material to be analyzed is taught on the template product by manual teaching.

[0147] S37: Determine the template guide trajectory according to each of the single material trajectories of the mechanism.

[0148] Specifically, the coordinate system of the mechanism single material trajectory is converted from the mechanism coordinate system of the manipulator to the image coordinate system of the visual tool, and the converted single material trajectories are associated to form a complete trajectory, which is used as the template guide trajectory.

[0149] A complete trajectory is a trajectory that contains only one starting point and one end point.

[0150] This embodiment determines the anchoring relationship between the feature points in the template feature point set and the template contour point detection frame according to the proximity principle, which provides a basis for subsequently determining the contour point detection frame of the deformed product to be processed based on the anchoring relationship between the template feature points and the detection frame.

[0151] In one embodiment, the step of determining the anchoring relationship between the template feature points and the detection frame according to each of the template contour point detection frames and the template feature point set comprises:

[0152] S351: acquiring any one of the template contour point detection frames from each of the template contour point detection frames as a detection frame to be anchored;

[0153] S352: Calculate the shortest distance between each feature point in the template feature point set and the detection frame to be anchored to obtain a single point shortest distance;

[0154] Specifically, the shortest distance between each feature point in the template feature point set and the detection frame to be anchored is calculated, and each calculated shortest distance is used as the single-point shortest distance.

[0155] S353: Find the single-point shortest distance with the smallest value from the single-point shortest distances as the target distance;

[0156] Specifically, the single-point shortest distance with the smallest value is found from the single-point shortest distances, and the found single-point shortest distance is used as the target distance.

[0157] S354: taking a point in the template feature point set corresponding to the target distance as an anchor point corresponding to the detection frame to be anchored;

[0158] Specifically, the point in the template feature point set corresponding to the target distance is used as the anchor point corresponding to the detection frame to be anchored, so that the feature point with the closest distance is used as the anchor point.

[0159] S355: taking the spatial position relationship between the anchor point corresponding to the detection frame to be anchored and the detection frame to be anchored as the single frame anchor relationship corresponding to the detection frame to be anchored;

[0160] Specifically, the spatial position relationship between the anchor point corresponding to the detection frame to be anchored and the detection frame to be anchored is used as the single frame anchor relationship corresponding to the detection frame to be anchored, thereby determining the anchor relationship between the detection frame to be anchored and the anchor point.

[0161] S356: Repeat the step of acquiring any one of the template contour point detection frames as the detection frame to be anchored, until the acquisition of the template contour point detection frames in the template contour point detection frames is completed;

[0162] Specifically, the step of obtaining any one of the template contour point detection frames as the detection frame to be anchored is repeated, that is, steps S351 to S356 are repeated until the acquisition of the template contour point detection frames in each of the template contour point detection frames is completed; when the acquisition of the template contour point detection frames in each of the template contour point detection frames is completed, it means that the single frame anchoring relationship of each of the template contour point detection frames is determined.

[0163] S357: Using each of the single frame anchoring relationships as the template feature point and detection frame anchoring relationship.

[0164] Specifically, each of the single frame anchoring relationships is used as the anchoring relationship between the template feature points and the detection frame, thereby providing a basis for subsequently determining the detection frame of the contour points to be processed of the deformed product to be processed based on the anchoring relationship between the template feature points and the detection frame.

[0165] This embodiment uses the feature point with the smallest shortest distance as the anchor point of the template contour point detection frame, and the spatial position relationship between the anchor point and the template contour point detection frame as a single frame anchor relationship. All single frame anchor relationships are used as the anchor relationship between the template feature point and the detection frame, thereby providing a basis for subsequently determining the contour point detection frame to be processed of the product to be processed based on the anchor relationship between the template feature point and the detection frame.

[0166] In one embodiment, the step of determining the template guide trajectory according to each of the mechanism single material trajectories includes:

[0167] S371: Obtaining the conversion relationship between the mechanism coordinates and the image coordinates corresponding to the template product;

[0168] Specifically, the conversion relationship between the mechanism coordinates and the image coordinates corresponding to the template product may be obtained from a database, or may be obtained from a third-party application.

[0169] S372: Mapping each of the mechanism single-material trajectories to an image coordinate system according to the conversion relationship to obtain an image single-material trajectory;

[0170] Specifically, according to the conversion relationship, the coordinates of each of the mechanism single-material trajectories are mapped from the mechanism coordinates of the manipulator to the image coordinate system of the visual tool, and the mapped trajectory is used as the image single-material trajectory.

[0171] S373: Based on the shortest path principle, the single-material trajectories of each image are spliced ​​to obtain the template guiding trajectory.

[0172] Specifically, based on the shortest path principle, the single-material trajectories of the images are spliced ​​to obtain a complete trajectory, and the complete trajectory is used as the template guide trajectory.

[0173] It can be understood that, based on the shortest path principle, when splicing the individual image single-material trajectories, if there are adjacent image single-material trajectories that do not intersect, auxiliary lines can be added to splice the individual image single-material trajectories into a complete trajectory.

[0174] In another embodiment of the present application, each of the single-material trajectories of the image may also be displayed, and the user may splice the displayed single-material trajectories of the image into a complete trajectory.

[0175] This embodiment splices the single-material trajectories of each image based on the shortest path principle to obtain the template guide trajectory, thereby making the path of the determined template guide trajectory the shortest, reducing the length of the determined template guide trajectory, and improving product processing efficiency.

[0176] like Figure 2 As shown, in one embodiment, the present application also proposes a 3D vision active guidance trajectory generation device for a multi-material product, the device comprising:

[0177] Point cloud acquisition module 801 is used to scan the product to be processed to obtain a point cloud to be processed;

[0178] The point cloud analysis module 802 is used to capture feature points of the point cloud to be processed to obtain a feature point set to be processed, and to extract regions of each material to be analyzed from the point cloud to be processed to obtain a single material region to be processed;

[0179] The data acquisition module 803 is used to acquire a template feature point set of a template product, an anchoring relationship between the template feature point and the detection frame, a single material contour point set of each template, and a template guide trajectory, wherein the template product and the product to be processed are the same type of product;

[0180] A detection frame determination module 804 for contour points to be processed is used to determine the detection frame of each contour point to be processed according to the feature point set to be processed, the template feature point set and the anchoring relationship between the template feature point and the detection frame;

[0181] The module 805 for determining the single-material contour point set to be processed is used to extract the single-material contour point set to be processed for each single-material region to be processed according to each of the single-material contour point detection frames to be processed;

[0182] The processing guide trajectory determination module 806 is used to adjust the template guide trajectory according to each of the processing single-material contour point sets and each of the template single-material contour point sets to obtain the processing guide trajectory corresponding to the processing product.

[0183] This embodiment first analyzes the feature point set to be processed and the single material area to be processed from the point cloud to be processed of the product to be processed, and then determines each contour point detection frame to be processed of the product to be processed based on the correspondence between the feature point set to be processed and the template feature point set and the anchoring relationship between the template feature point and the detection frame, and then extracts the single material contour point set to be processed for each single material area to be processed according to each contour point detection frame to be processed, and finally adjusts the template guide trajectory according to each single material contour point set to be processed and each template single material contour point set to determine the guide trajectory to be processed suitable for the product to be processed, thereby generating a guide trajectory for a multi-material product, providing a basis for automated dispensing or spraying operations for multi-material products, and improving production efficiency; and adjusting the template guide trajectory according to each single material contour point set to be processed and each template single material contour point set, fully considering the deformation of the product to be processed, improving the accuracy of the determined guide trajectory to be processed, and improving the effect of automated dispensing or spraying operations.

[0184] Figure 3 FIG. 1 shows an internal structure diagram of a computer device in an embodiment. The computer device may be a terminal or a server. Figure 3 As shown, the computer device includes a processor, a memory and a network interface connected via a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor may implement a method for generating a 3D visual active guidance trajectory for a multi-material product. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor may execute a method for generating a 3D visual active guidance trajectory for a multi-material product. Those skilled in the art will appreciate that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0185] In one embodiment, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the following steps:

[0186] Scan the product to be processed to obtain a point cloud to be processed;

[0187] Capturing feature points of the point cloud to be processed to obtain a feature point set to be processed, and extracting regions of each material to be analyzed from the point cloud to obtain a single material region to be processed;

[0188] Acquire a template feature point set of a template product, an anchoring relationship between the template feature point and a detection frame, a single-material contour point set of each template, and a template guide trajectory, wherein the template product and the product to be processed are the same type of product;

[0189] Determine the detection frame of each contour point to be processed according to the feature point set to be processed, the template feature point set and the anchoring relationship between the template feature point and the detection frame;

[0190] Extracting a set of single-material contour points to be processed for each single-material area to be processed according to each of the contour point detection frames to be processed;

[0191] According to each of the to-be-processed single-material contour point sets and each of the template single-material contour point sets, the template guide trajectory is adjusted to obtain the to-be-processed guide trajectory corresponding to the to-be-processed product.

[0192] This embodiment first analyzes the feature point set to be processed and the single material area to be processed from the point cloud to be processed of the product to be processed, and then determines each contour point detection frame to be processed of the product to be processed based on the correspondence between the feature point set to be processed and the template feature point set and the anchoring relationship between the template feature point and the detection frame, and then extracts the single material contour point set to be processed for each single material area to be processed according to each contour point detection frame to be processed, and finally adjusts the template guide trajectory according to each single material contour point set to be processed and each template single material contour point set to determine the guide trajectory to be processed suitable for the product to be processed, thereby generating a guide trajectory for a multi-material product, providing a basis for automated dispensing or spraying operations for multi-material products, and improving production efficiency; and adjusting the template guide trajectory according to each single material contour point set to be processed and each template single material contour point set, fully considering the deformation of the product to be processed, improving the accuracy of the determined guide trajectory to be processed, and improving the effect of automated dispensing or spraying operations.

[0193] In one embodiment, a computer-readable storage medium is provided, storing a computer program, wherein when the computer program is executed by a processor, the processor performs the following steps:

[0194] Scan the product to be processed to obtain a point cloud to be processed;

[0195] Capturing feature points of the point cloud to be processed to obtain a feature point set to be processed, and extracting regions of each material to be analyzed from the point cloud to obtain a single material region to be processed;

[0196] Acquire a template feature point set of a template product, an anchoring relationship between the template feature point and a detection frame, a single-material contour point set of each template, and a template guide trajectory, wherein the template product and the product to be processed are the same type of product;

[0197] Determine the detection frame of each contour point to be processed according to the feature point set to be processed, the template feature point set and the anchoring relationship between the template feature point and the detection frame;

[0198] Extracting a set of single-material contour points to be processed for each single-material area to be processed according to each of the contour point detection frames to be processed;

[0199] According to each of the to-be-processed single-material contour point sets and each of the template single-material contour point sets, the template guide trajectory is adjusted to obtain the to-be-processed guide trajectory corresponding to the to-be-processed product.

[0200] This embodiment first analyzes the feature point set to be processed and the single material area to be processed from the point cloud to be processed of the product to be processed, and then determines each contour point detection frame to be processed of the product to be processed based on the correspondence between the feature point set to be processed and the template feature point set and the anchoring relationship between the template feature point and the detection frame, and then extracts the single material contour point set to be processed for each single material area to be processed according to each contour point detection frame to be processed, and finally adjusts the template guide trajectory according to each single material contour point set to be processed and each template single material contour point set to determine the guide trajectory to be processed suitable for the product to be processed, thereby generating a guide trajectory for a multi-material product, providing a basis for automated dispensing or spraying operations for multi-material products, and improving production efficiency; and adjusting the template guide trajectory according to each single material contour point set to be processed and each template single material contour point set, fully considering the deformation of the product to be processed, improving the accuracy of the determined guide trajectory to be processed, and improving the effect of automated dispensing or spraying operations.

[0201] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate

[0202] SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAM bus dynamic RAM (RDRAM), etc.

[0203] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0204] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A 3D vision active guidance trajectory generation method for multi-material products, the method include: Scan the product to be processed to obtain a point cloud to be processed; Capturing feature points of the point cloud to be processed to obtain a feature point set to be processed, and extracting regions of each material to be analyzed from the point cloud to obtain a single material region to be processed; Acquire a template feature point set of a template product, an anchoring relationship between the template feature point and a detection frame, a single-material contour point set of each template, and a template guide trajectory, wherein the template product and the product to be processed are the same type of product; Determine the detection frame of each contour point to be processed according to the feature point set to be processed, the template feature point set and the anchoring relationship between the template feature point and the detection frame; Extracting a set of single-material contour points to be processed for each single-material area to be processed according to each of the contour point detection frames to be processed; According to each of the to-be-processed single-material contour point sets and each of the template single-material contour point sets, the template guide trajectory is adjusted to obtain the to-be-processed guide trajectory corresponding to the to-be-processed product.

2. The method for generating 3D visual active guidance trajectories for multi-material products according to claim 1, It is characterized in that The step of capturing feature points of the point cloud to be processed to obtain a set of feature points to be processed includes: Inputting the to-be-processed point cloud into a target detection model corresponding to the template product to detect fixed features, and obtaining each fixed feature detection frame; Searching for a center point within a range corresponding to each of the fixed feature detection frames in the point cloud to be processed as a single feature center point; Each of the single feature center points is used as the feature point set to be processed.

3. The method for generating 3D visual active guidance trajectories for multi-material products according to claim 1, It is characterized in that The step of extracting the region of each material to be analyzed from the point cloud to be processed to obtain the single material region to be processed comprises: Using a preset filter combination method, filtering the point cloud to be processed to obtain a point cloud to be extracted, wherein the filter combination method is a method obtained by combining at least one of mean filtering, median filtering, Gaussian filtering and cluster filtering; The point cloud to be extracted is input into a region segmentation model corresponding to the template product to perform region segmentation corresponding to each material to be analyzed, so as to obtain the single material region to be processed.

4. The method for generating 3D visual active guidance trajectories for multi-material products according to claim 1, It is characterized in that The step of extracting a set of single-material contour points to be processed for each single-material area to be processed according to each of the contour point detection frames to be processed comprises: Searching for boundary extreme value points within a range corresponding to each of the to-be-processed contour point detection boxes corresponding to the first material in the to-be-processed single material area corresponding to the first material, to obtain to-be-processed single material contour points, wherein the first material is any one of the various to-be-analyzed materials; Each of the to-be-processed single-material contour points corresponding to the first material is used as the to-be-processed single-material contour point set corresponding to the first material.

5. The method for generating 3D visual active guidance trajectories for multi-material products according to claim 1, It is characterized in that Before the step of obtaining the template feature point set of the template product, the anchoring relationship between the template feature point and the detection frame, the single material contour point set of each template and the template guide trajectory, the step also includes: Scanning the template product to obtain a template point cloud; Capturing feature points of the template point cloud to obtain the template feature point set, and extracting regions of each material to be analyzed from the template point cloud to obtain a template single material region; Acquire each template contour point detection frame input by the user according to each template single material area; According to each of the template contour point detection frames corresponding to the second material, extracting the template single-material contour point set for the template single-material region corresponding to the second material, wherein the second material is any one of the various materials to be analyzed; Determining the anchoring relationship between the template feature points and the detection frame according to each of the template contour point detection frames and the template feature point set; Teaching the single material trajectory of the mechanism corresponding to each material to be analyzed on the template product; The template guide trajectory is determined according to the single material trajectory of each mechanism.

6. The method for generating 3D visual active guidance trajectories for multi-material products according to claim 5, It is characterized in that The step of determining the anchoring relationship between the template feature points and the detection frame according to each of the template contour point detection frames and the template feature point set comprises: Acquire any one of the template contour point detection frames from each of the template contour point detection frames as a detection frame to be anchored; Calculate the shortest distance between each feature point in the template feature point set and the detection frame to be anchored to obtain the shortest distance of a single point; Find the shortest distance at the single point with the smallest value from among the shortest distances at the single point, and use it as the target distance; Taking the point in the template feature point set corresponding to the target distance as the anchor point corresponding to the detection frame to be anchored; The spatial position relationship between the anchor point corresponding to the detection frame to be anchored and the detection frame to be anchored is used as the single frame anchor relationship corresponding to the detection frame to be anchored; Repeat the step of acquiring any one of the template contour point detection frames as the detection frame to be anchored, until the acquisition of the template contour point detection frames in the template contour point detection frames is completed; Each of the single frame anchoring relationships is used as the anchoring relationship between the template feature point and the detection frame.

7. The method for generating 3D vision active guidance trajectory for multi-material products according to claim 5, It is characterized in that The step of determining the template guide trajectory according to each of the mechanism single material trajectories comprises: Obtaining the conversion relationship between the mechanism coordinates and the image coordinates corresponding to the template product; According to the conversion relationship, each of the mechanism single material trajectories is mapped to the image coordinate system to obtain the image single material trajectory; Based on the shortest path principle, the single-material trajectories of each image are spliced ​​to obtain the template guiding trajectory.

8. A 3D vision active guidance trajectory generation device for multi-material products. It is characterized in that The device comprises: Point cloud acquisition module, used to scan the product to be processed and obtain the point cloud to be processed; A point cloud analysis module is used to capture feature points of the point cloud to be processed to obtain a feature point set to be processed, and to extract regions of each material to be analyzed from the point cloud to be processed to obtain a single material region to be processed; A data acquisition module, used to acquire a template feature point set of a template product, an anchoring relationship between the template feature point and the detection frame, a single-material contour point set of each template, and a template guide trajectory, wherein the template product and the product to be processed are the same type of product; A module for determining a detection frame of contour points to be processed, used to determine the detection frame of each contour point to be processed according to the set of feature points to be processed, the set of template feature points and the anchoring relationship between the template feature points and the detection frame; A module for determining a single-material contour point set to be processed, configured to extract a single-material contour point set to be processed for each single-material region to be processed according to each of the single-material contour point detection frames to be processed; The module for determining the guide trajectory to be processed is used to adjust the template guide trajectory according to each of the single-material contour point sets to be processed and each of the single-material contour point sets of the template to obtain the guide trajectory to be processed corresponding to the product to be processed.

9. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 7.

Images