Carriage positioning and material detection method, medium and equipment

Abstract

The invention relates to a carriage positioning and material detection method, a medium and equipment. Corner point coordinates and edge contours of the upper surface and the bottom of a compartment under a mechanical arm tail end coordinate system are obtained by obtaining a transformation matrix between an acquisition equipment coordinate system and the mechanical arm tail end coordinate system, initial point cloud data are generated according to prior acquisition equipment internal reference and compartment depth map information and are segmented, compartment point cloud data are obtained, and the compartment point cloud data are stored in a storage box. And segmenting the carriage point cloud data to obtain a plurality of planes and corresponding normal vectors, obtaining material point clouds according to the normal vectors of the planes, obtaining bounding boxes of the material point clouds and coordinates of the bounding boxes according to the carriage depth map, and obtaining sizes and coordinates of the bounding boxes under the mechanical arm tail end coordinate system according to the conversion matrix. The problems of insufficient adaptability, low accuracy, insufficient efficiency and the like during carriage positioning and size detection in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a carriage positioning and material detection method, medium and equipment. Background Art

[0002] In traditional automatic loading systems, the acquisition of carriage size and position usually relies on manual measurement or radar detection devices. Manual measurement methods are inefficient, prone to errors, and have poor adaptability when carriage size and loading conditions change. Although radar detection technology improves accuracy and reduces costs, it has deficiencies in the precise identification of carriage corners and the dynamic adjustment of loading positions, resulting in limited adaptability when the length, parking position and angle of the carriage change greatly.

[0003] Therefore, how to improve the existing technology's lack of adaptability, low accuracy, and insufficient efficiency in car positioning and size detection is a technical problem that needs to be urgently solved in this field. Summary of the invention

[0004] Based on this, the purpose of this application is to provide a car positioning and material detection method, medium and equipment to solve at least one technical problem mentioned in the above background technology.

[0005] In a first aspect, the present application provides a carriage positioning and material detection method, comprising:

[0006] Obtain the transformation matrix between the acquisition device coordinate system and the robot end coordinate system;

[0007] Control the robot arm to move in the set direction, collect the depth map of the car body, and obtain the coordinates of the corner points on the upper and bottom surfaces of the car body in the coordinate system of the robot arm end;

[0008] Generate initial point cloud data based on the prior acquisition device internal parameters and the car depth map information, and segment the car according to the corner points of the upper surface and the bottom of the car to obtain the car point cloud data;

[0009] The car body point cloud data is segmented and plane fitting is performed to obtain several planes and corresponding normal vectors, so as to obtain the material point cloud according to the normal vectors of each plane;

[0010] Obtain the minimum external figure of each material point cloud to obtain the bounding box of each material point cloud and the coordinates of each bounding box according to the car depth map;

[0011] The coordinates of each bounding box are converted to the robot end coordinate system according to the transformation matrix, and the size and coordinates of each bounding box in the robot end coordinate system are obtained.

[0012] Furthermore, the step of obtaining the transformation matrix includes:

[0013] S11: Install the calibration plate at the set position, control the acquisition device to scan the surface of the calibration plate, and obtain the coordinates and numbers corresponding to each feature point in the acquisition device coordinate system;

[0014] S12: Install the calibration device at the end of the robot arm, and touch the feature points on the surface of the calibration plate in turn to obtain the coordinates and numbers corresponding to each feature point in the coordinate system of the end of the robot arm;

[0015] S13: constructing a number of feature point pairs according to the numbers corresponding to the feature points, and establishing a linear equation system according to the feature point pairs;

[0016] S14: Substitute the coordinates corresponding to each feature point into the linear equation group, and solve it to obtain the transformation matrix between the acquisition device coordinate system and the robot end coordinate system.

[0017] S15: Set a number of verification points, and obtain the coordinates and numbers of each verification point in the acquisition device coordinate system and the robot end coordinate system;

[0018] S16: Convert all verification points to the same coordinate system, obtain the distance between verification points with the same number, and determine whether it is less than the distance threshold. If so, the conversion matrix is ​​qualified. If not, return to step S11.

[0019] Furthermore, the step of obtaining the coordinates of the corner points on the upper surface and the bottom of the carriage in the robot arm end coordinate system includes:

[0020] respectively collecting a carriage depth map of a first area of ​​the carriage and a carriage depth map of a second area of ​​the carriage;

[0021] Perform gradient calculation on the depth map of each carriage to obtain the corresponding carriage contour image;

[0022] The edge contour of the carriage is obtained according to the contour images of each carriage, so as to obtain the geometric data of the carriage according to the edge contour of the carriage and make a corresponding carriage template;

[0023] Perform template matching on the carriage depth map according to each carriage template to obtain the coordinates of the corner points on the upper surface of the carriage, and map them to the bottom of the carriage according to the carriage depth map to obtain the coordinates of the corner points on the bottom of the carriage;

[0024] According to the conversion matrix between the acquisition device coordinate system and the robot end coordinate system, the coordinates of the corner points on the upper surface and bottom of the carriage in the robot base coordinate system are obtained.

[0025] Furthermore, the step of obtaining the coordinates of the corner points on the upper surface and the bottom of the carriage in the robot arm end coordinate system also includes:

[0026] Dedistort the car depth map to obtain the corrected corner point coordinates;

[0027] According to the prior carriage height, the carriage depth map is filtered to obtain an optimized carriage depth map.

[0028] Furthermore, the step of obtaining the coordinates of the corner points on the upper surface and the bottom of the carriage in the robot arm end coordinate system also includes:

[0029] Scan the surface of the carriage at set intervals to obtain several sets of carriage depth maps;

[0030] Perform gradient calculation on the depth map of each carriage to obtain the corresponding carriage contour image;

[0031] A corresponding carriage template is made according to each carriage contour image, so as to perform template matching on the carriage depth map according to each carriage template to obtain the coordinates of the corner points on the upper surface of the carriage;

[0032] According to the depth map of the car body, the coordinates of the corner points on the upper surface of the car body are mapped to the bottom of the car body to obtain the coordinates of the corner points on the bottom of the car body. According to the transformation matrix between the coordinate system of the acquisition device and the coordinate system of the end of the robotic arm, the coordinates of the corner points on the upper surface and bottom of the car body in the robot base coordinate system are obtained.

[0033] Furthermore, the step of making corresponding carriage templates according to the carriage contour images, performing template matching on the carriage depth map according to the carriage templates, and obtaining the coordinates of the corner points on the upper surface of the carriage includes:

[0034] Obtaining a carriage edge contour according to each carriage contour image, so as to obtain carriage geometry data according to the carriage edge contour;

[0035] According to the geometric data of the carriage, a corresponding carriage template is made, and the carriage depth map is template matched according to each carriage template to obtain the coordinates of the corner points of the front half of the carriage;

[0036] According to the direction and distance between the prior midpoint of the carriage and the corner coordinates of the front half of the carriage, the corner coordinates of the rear of the carriage are reversely deduced to obtain the corner coordinates of the upper surface of the carriage.

[0037] Further, the step of obtaining the car point cloud data includes:

[0038] According to the prior acquisition equipment internal parameters and the depth value of each pixel in the car depth map, the three-dimensional coordinates corresponding to each pixel are obtained to generate initial point cloud data;

[0039] Get the three-dimensional coordinates of all corner points and get the extreme values ​​of the corner point coordinates;

[0040] A filtering range is constructed according to the extreme values ​​of the corner point coordinates, so as to perform a filtering operation on the initial point cloud data according to the filtering range to obtain the carriage point cloud data.

[0041] Further, the steps of obtaining the material point cloud include:

[0042] Segment the carriage point cloud data to obtain several point cloud clusters;

[0043] Perform plane fitting on each point cloud cluster to obtain several planes and corresponding normal vectors;

[0044] Get the angle between each normal vector and the z-axis to determine whether it is greater than the set angle threshold. If not, keep it unchanged. If so, remove all point clouds corresponding to the plane to obtain the material point cloud.

[0045] In a second aspect, the present application further provides a computer storage medium storing an executable program code; the executable program code is used to execute the carriage positioning and material detection method described in any one of the first aspects.

[0046] In a third aspect, the present application also provides a terminal device, including a memory and a processor; the memory stores a program code that can be executed by the processor; the program code is used to execute the carriage positioning and material detection method described in any one of the first aspects.

[0047] The present invention provides a method, medium and device for car compartment positioning and material detection, which obtains the conversion matrix between the acquisition device coordinate system and the mechanical arm end coordinate system, and then controls the mechanical arm to move in a set direction, collects the car compartment depth map, and obtains the corner point coordinates and edge contours of the upper surface and bottom of the car compartment in the mechanical arm end coordinate system; the car compartment depth map, including the car compartment depth map and the car compartment depth map, thereby generating initial point cloud data according to the prior acquisition device internal parameters and the car compartment depth map information, and segmenting the car compartment upper surface and the car compartment bottom according to the corner points to obtain the car compartment point cloud data, complete the precise positioning of the car compartment, segment the car compartment point cloud data, and perform plane fitting. Combine to obtain several planes and corresponding normal vectors, so as to obtain material point clouds according to the normal vectors of each plane, obtain the minimum external graphics of each material point cloud, obtain the bounding box of each material point cloud and the coordinates of each bounding box according to the depth map of the carriage, convert the coordinates of each bounding box to the coordinate system of the end of the robot arm according to the transformation matrix, obtain the size and coordinates of each bounding box in the coordinate system of the end of the robot arm, generate an accurate model of the carriage by collecting the depth image and depth data of the carriage, and then extract the actual position and size information of the material from the inside of the carriage through point cloud processing and image processing technology, thereby enhancing the adaptability of the existing technology in different environments and the accuracy of material recognition. The existing technology solves the problems of insufficient adaptability and low accuracy in carriage positioning and size detection, as well as insufficient efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of a method for carriage positioning and material detection according to an embodiment of the present invention;

[0049] Figure 2Schematic diagram of car point cloud data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] It should be noted that if the embodiments of the present invention involve directional indications, such as up, down, left, right, front, back, etc., then the directional indication is only used to explain the relative position relationship, movement status, etc. between the components in a certain specific posture. If the specific posture changes, the directional indication will also change accordingly. In addition, if the embodiments of the present invention involve descriptions of "first, second", "S1, S2", "step one, step two", etc., then such descriptions are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of indicated technical features or indicating the execution order of the method, etc. Those skilled in the art can understand that anything that does not violate the gist of the invention under the technical concept of the invention should be included in the protection scope of the present invention.

[0052] like Figure 1 As shown, the present invention provides a carriage positioning and material detection method:

[0053] S1: Obtain the transformation matrix between the acquisition device coordinate system and the robot end coordinate system;

[0054] Specifically, it is optional but not limited to installing the acquisition device at the end of the robotic arm, and installing a calibration plate at a set position, obtaining the coordinates of each feature point on the calibration plate in the acquisition device coordinate system and the robotic arm end coordinate system, forming several groups of feature point pairs, and constructing and solving a linear equation system, thereby calculating the transformation matrix between the acquisition device coordinate system and the robotic arm end coordinate system.

[0055] For example, a material detection system can be optionally provided, including two 3D structured light cameras and a truss system with two truss arms, the truss coordinate system takes the collective center point of the trusses as the origin, a gripper or a suction cup is provided at the end of the truss arm, and the 3D structured light camera is installed in the gripper or the suction cup.

[0056] Preferably, the step of obtaining a conversion matrix between the acquisition device coordinate system and the robot arm end coordinate system may optionally include:

[0057] S11: Install the calibration plate at the set position, control the acquisition device to scan the surface of the calibration plate, and obtain the coordinates and numbers corresponding to each feature point in the acquisition device coordinate system;

[0058] Specifically, it is optional but not limited to installing the calibration plate at a set position, and then scanning the surface of the calibration plate by the acquisition device on the robotic arm to obtain the surface image data of the calibration plate to obtain the coordinates of each feature point on the surface of the calibration plate, and obtain the number of each feature point according to the prior area of ​​interest; the specific shape and size of the calibration plate are arbitrarily set by technical personnel in this field; the set position is arbitrarily set by technical personnel in this field, and it is necessary to enable the acquisition device to clearly acquire the surface data of the calibration plate, and the calibration device at the end of the robotic arm can touch each feature point on the surface of the calibration plate; the surface image data may optionally include a car texture image and a car depth map; the prior area of ​​interest is arbitrarily set by technical personnel in this field.

[0059] For example, the car texture image and car depth map of the calibration plate surface can be obtained by the acquisition device, so as to obtain the plane coordinates of all feature points according to the car texture image, and then obtain the 3D coordinates of all feature points according to the car depth map. in i is the feature point number, and n is the number of feature points.

[0060] S12: Install the calibration device at the end of the robot arm, and touch the feature points on the surface of the calibration plate in turn to obtain the coordinates and numbers corresponding to each feature point in the coordinate system of the end of the robot arm;

[0061] For example, taking the calibration needle as an example, the calibration needle can be optionally installed at the end of the robotic arm, and then the technicians in this field control the calibration needle to touch the feature points on the surface of the calibration plate in turn, and record the coordinates and numbers of each feature point. in i is the feature point number, and n is the number of feature points.

[0062] S13: constructing a number of feature point pairs according to the numbers corresponding to the feature points, and establishing a linear equation system according to the feature point pairs;

[0063] S14: Substitute the coordinates corresponding to each feature point into the linear equation group, and solve it to obtain the transformation matrix between the acquisition device coordinate system and the robot end coordinate system.

[0064] Specifically, it is optional but not limited to taking feature points with the same number as a pair of feature point pairs, traversing all feature points, and obtaining several feature point pairs to construct equation 1-1 for each point pair:

[0065]

[0066] Then expand the equations of each point pair to obtain the linear equation system 1-2:

[0067]

[0068] in, is the i-th feature point in the robot end coordinate system, t ct and R ct are the translation vector and rotation matrix between the acquisition device coordinate system and the robot end coordinate system, is the i-th feature point in the acquisition device coordinate system.

[0069] It is preferred to solve it according to the least squares method to obtain the translation vector t between the acquisition device coordinate system and the robot end coordinate system ct and the rotation matrix R ct ;t ct and R ct Together they form the transformation matrix between the two.

[0070] Preferably, since there may be factors such as process conditions and environmental influences during the calibration process, which may lead to deviations in the calibration results, it is necessary to verify the conversion matrix obtained by calibration. Therefore, after step S14, the following is further included:

[0071] S15: Set a number of verification points, and obtain the coordinates and numbers of each verification point in the acquisition device coordinate system and the robot end coordinate system;

[0072] S16: Convert all verification points to the same coordinate system, obtain the distance between verification points with the same number, and determine whether it is less than the distance threshold. If so, the conversion matrix is ​​qualified. If not, return to step S11.

[0073] Specifically, it is optional but not limited to that a technician in this field sets any number of point clouds on the calibration plate as verification points and sets a distance threshold, and then obtains the coordinates and numbers of all verification points in the acquisition device coordinate system and the robot end coordinate system according to steps S11 and S12, and then converts all verification points from the acquisition device coordinate system to the robot end coordinate system or from the robot end coordinate system to the acquisition device coordinate system according to the conversion matrix obtained in step S14, and obtains the distance between verification points with the same number, and determines whether it is greater than the distance threshold. If not, it means that the error of the conversion matrix obtained in step S14 is low and meets the requirements, and subsequent steps can be performed. If yes, it means that the error between the conversion matrices is large, and it is necessary to reset the parameters such as the calibration plate position, and return to step S11 for recalibration until the iteration condition is met; the distance threshold and iteration condition are arbitrarily set by a technician in this field. Preferably, the iteration condition can be selected as the number of iterations.

[0074] S2: Control the robot arm to move in the set direction, collect the depth map of the car body, and obtain the coordinates of the corner points on the upper surface and bottom of the car body in the coordinate system of the end of the robot arm;

[0075] Specifically, it is optional but not limited to controlling the robotic arm to move in a set direction, scanning the upper surface of the car, obtaining a car depth map of the upper surface of the car, so as to obtain the corner points and edge contours of the upper surface of the car, and mapping the corner points and edge contours of the upper surface of the car to the bottom of the car according to the car depth map, so as to obtain the corner point coordinates and edge contours of the upper and bottom surfaces of the car in the coordinate system of the robotic arm end.

[0076] Preferably, taking the material detection system in step S1 as an example, the specific steps of S2 may optionally include:

[0077] S21: Collecting a depth map of the first area of ​​the carriage and a depth map of the second area of ​​the carriage respectively;

[0078] Specifically, the first robotic arm can be optionally controlled to scan the first area of ​​the car to obtain a car depth map of the first area of ​​the car, and the second robotic arm can be simultaneously controlled to scan the second area of ​​the car to obtain a car depth map of the second area of ​​the car; the first area of ​​the car and the second area of ​​the car can be optionally set by technical personnel in this field.

[0079] Preferably, the first area of ​​the carriage is the front half of the carriage near the front of the vehicle, and the second area of ​​the carriage is the rear half of the carriage near the rear of the vehicle.

[0080] S22: performing gradient calculation on the depth map of each carriage to obtain a corresponding carriage contour image;

[0081] Specifically, it is optional but not limited to using the Sobel operator to calculate the gradient maps of the car depth map of the first area and the second area of ​​the car in the X and Y directions, so as to respectively obtain the edge features in the horizontal and vertical directions of the image, more clearly display the edge information of the image, and obtain the car contour image corresponding to the first area and the second area of ​​the car; the image contrast enhancement algorithm may optionally include commonly used image contrast enhancement algorithms such as histogram equalization algorithm, contrast stretching algorithm, contrast limited adaptive histogram equalization algorithm and gamma correction algorithm.

[0082] S23: Obtaining a carriage edge contour according to each carriage contour image, obtaining carriage geometry data according to the carriage edge contour, and making a corresponding carriage template;

[0083] Specifically, it is optional but not limited to using a contour detection algorithm to obtain the car contour in the car contour map, and calculate the car geometry data based on the obtained car contour, so as to make car templates for the first area and the second area of ​​the car based on the car geometry data; the contour detection algorithm may optionally include common algorithms such as Canny edge detection, Sobel operator, edge tracking method, etc.; the car geometry data may optionally include geometric data such as car width and diagonal length.

[0084] S24: performing template matching on the carriage depth map according to each carriage template to obtain the coordinates of the corner points on the upper surface of the carriage, and mapping them to the bottom of the carriage according to the carriage depth map to obtain the coordinates of the corner points on the bottom of the carriage;

[0085] S25: According to the conversion matrix between the acquisition device coordinate system and the robot end coordinate system, the coordinates of the corner points of the upper surface and the bottom of the carriage in the robot base coordinate system are obtained.

[0086] Specifically, it is optional but not limited to using each car template to perform template matching on the car depth map to obtain the corner point coordinates and edge contours of the upper surface of the first area and the second area of ​​the car, and then according to the car height information in the car depth map, the corner point coordinates and edge contours of the upper surface of the car are mapped to the bottom of the car to obtain the corner point coordinates and edge contours of the bottom of the car, and then according to the transformation matrix between the acquisition device coordinate system and the robot arm end coordinate system obtained in step S1, it is converted to the robot base coordinate system.

[0087] More specifically, during the acquisition process, the acquisition device may be affected by various factors, such as a wide-angle lens causing points near the center of the image to be stretched or compressed, or an imperfect alignment between the lens and the image sensor causing the image to be tilted or offset, etc., so that the coordinates of each point in the acquired image are inconsistent with the actual position. Therefore, after obtaining the depth map within the acquisition range of the acquisition device in step S21, it is preferred that the following further be included:

[0088] S21': Dedistort the depth map to obtain the corrected corner point coordinates;

[0089] Specifically, it is optional but not limited to using an image dedistortion method to dedistort the car depth map based on parameters such as focal length, principal point and distortion coefficient in the prior acquisition device internal parameters, so as to obtain the corrected corner point coordinates; the image dedistortion method may include common algorithms such as OpenCV dedistortion function, remapping method, deep learning method and geometric correction method.

[0090] S22': Filter the carriage depth map according to the prior carriage height to obtain an optimized carriage depth map.

[0091] Specifically, it is optional but not limited to setting a height threshold range based on the prior car height, and then obtaining the depth of each pixel in the car depth map to determine whether it is within the height threshold range. If so, it proves that the point is part of the car and needs to be retained. If not, it means that the point is noise or irrelevant background information and needs to be eliminated, so as to obtain an optimized car depth map to remove interference and improve recognition accuracy; the prior car height can be optionally obtained based on field measurements, vehicle specification databases, etc.; the height threshold range is arbitrarily set by personnel in this field.

[0092] Preferably, due to factors such as process conditions and environmental influences, it may be possible to operate only a single robotic arm when collecting the vehicle depth map. Therefore, step S2 may also optionally include:

[0093] S26: Scan the surface of the carriage at set intervals to obtain several sets of carriage depth maps;

[0094] Specifically, it is optional but not limited to setting the collection distance, then controlling the robotic arm to move the collection distance each time, and then scanning the surface of the car through the collection device, and each scan can obtain a set of car depth maps; the collection distance is arbitrarily set by technical personnel in this field.

[0095] S27: performing gradient calculation on the depth map of each carriage to obtain a corresponding carriage contour image;

[0096] Specifically, it is optional to obtain a number of corresponding carriage contour images according to step S22.

[0097] S28: creating corresponding carriage templates according to the carriage contour images, and performing template matching on the carriage depth map according to the carriage templates to obtain the coordinates of the corner points on the upper surface of the carriage;

[0098] Specifically, it is optional but not limited to using a contour detection algorithm to obtain the car contour in the car contour map, and calculate the car geometry data based on the obtained car contour, so as to make car templates for various areas of the car based on the car geometry data; the contour detection algorithm may optionally include common algorithms such as Canny edge detection, Sobel operator, edge tracking method, etc.; the car geometry data may optionally include geometric data such as car width and diagonal length.

[0099] For example, taking the carriage templates of the carriage head and the middle of the carriage as an example, the carriage edge contour is obtained according to each carriage contour image to obtain the carriage geometry data, and the steps of making the corresponding carriage template may optionally include:

[0100] S281: Obtaining a carriage edge contour according to each carriage contour image, so as to obtain carriage geometry data according to the carriage edge contour;

[0101] Specifically, the carriage edge contour in each carriage contour image can be obtained according to step S23 as the initial carriage edge contour, so as to calculate the distance between the carriage edge contours in each carriage contour image as the carriage width.

[0102] S282: creating corresponding carriage templates according to the carriage geometry data, and performing template matching on the carriage depth map according to each carriage template to obtain the coordinates of the corner points of the front half of the carriage;

[0103] S283: According to the direction and distance between the a priori midpoint of the carriage and the corner point coordinates of the front half of the carriage, the corner point coordinates of the rear of the carriage are reversely derived to obtain the corner point coordinates of the upper surface of the carriage.

[0104] Specifically, the coordinates of the corner points of the front half of the car, such as the head and the middle of the car, can be obtained according to step S24, and then the coordinates of the corner points of the rear of the car can be reversely deduced based on the direction and distance between the a priori coordinates of the corner points of the midpoint of the car and the head of the car, as well as the symmetrical structure of the car itself, to obtain the coordinates of the corner points of the upper surface of the car; the a priori midpoint of the car can be optionally obtained based on field measurements, vehicle specification database, etc.

[0105] S29: According to the depth map of the car body, the coordinates of the corner points on the upper surface of the car body are mapped to the bottom of the car body to obtain the coordinates of the corner points on the bottom of the car body, and according to the conversion matrix between the coordinate system of the acquisition device and the coordinate system of the end of the robot arm, the coordinates of the corner points on the upper surface and bottom of the car body in the robot base coordinate system are obtained.

[0106] Specifically, the coordinates of the corner points of the bottom of the carriage may be obtained according to step S24, and then the coordinates of the corner points of the upper surface and the bottom of the carriage in the robot base coordinate system may be obtained according to step S25.

[0107] Preferably, when operating a single robotic arm to collect the depth map of the car body, due to the influence of the structure of the machine itself, it is impossible to directly collect the depth map of the entire car body, such as only the depth map of the head and the middle of the car body can be collected, or only the depth map of the tail and the middle of the car body can be collected. It is necessary to map half of the obtained car body corner points to the other half according to the symmetrical structure of the car body itself, so as to obtain all the car body corner points.

[0108] S3: Generate initial point cloud data based on the prior acquisition device internal parameters and the car depth map information, and segment the car according to the corner points of the upper surface and the bottom of the car to obtain the car point cloud data;

[0109] Specifically, but not limited to, based on the internal parameters of the prior acquisition device and the depth information of each pixel in the carriage depth map, the three-dimensional coordinates corresponding to each pixel are obtained, so as to obtain the initial point cloud data corresponding to the carriage. Then, a filtering range is constructed according to the corner points on the upper surface and the bottom of the carriage, and the pass-through filtering is performed on the initial point cloud data to remove the noise or background point cloud in the initial point cloud data, and the carriage point cloud data as shown in Figure 2 can be obtained; the internal parameters of the prior acquisition device can be pre-calibrated by those skilled in the art or obtained according to technical documents.

[0110] Preferably, the steps of generating the initial point cloud data according to the internal parameters of the prior acquisition device and the carriage depth map information and segmenting it according to the corner points on the upper surface and the bottom of the carriage to obtain the carriage point cloud data may include:

[0111] S31: According to the internal parameters of the prior acquisition device and the depth values of each pixel in the carriage depth map, obtain the three-dimensional coordinates corresponding to each pixel and generate the initial point cloud data;

[0112] Exemplarily, the internal parameter matrix of the acquisition device can be obtained, including the focal lengths (fx, fy) and the principal point coordinates (cx, cy). Then, each pixel in the carriage depth map is traversed, and its depth value Z is extracted, so as to calculate the three-dimensional coordinates corresponding to each pixel according to the coordinate calculation formula to generate the initial point cloud data.

[0113] Preferably, the coordinate calculation formula can be optionally expressed as Equations 3-1 and 3-2:

[0114]

[0115] where (X j , Y j , Z j ) are the three-dimensional coordinates corresponding to the j-th pixel, 0 < j ≤ M, M is the number of pixels, (u j , v j ) are the coordinates of the j-th pixel in the depth map, (fx, fy) are the focal lengths in the internal parameter matrix of the acquisition device, and (cx, cy) are the principal point coordinates in the internal parameter matrix of the acquisition device.

[0116] S32: Obtain the three-dimensional coordinates of all corner points to obtain the extreme values of the corner point coordinates;

[0117] S33: Construct a filtering range according to the extreme values of the corner point coordinates, and perform a filtering operation on the initial point cloud data according to the filtering range to obtain the carriage point cloud data.

[0118] Specifically, you can optionally obtain the three-dimensional coordinates of all corner points on the upper surface and bottom of the car, and get the extreme values ​​of the coordinates of each corner point on the x-axis, y-axis, and z-axis. You can then build a filtering range based on the extreme values ​​of the coordinates, and then remove the point clouds in the initial point cloud data whose coordinates are not within the filtering range, thereby obtaining the car point cloud data.

[0119] For example, the coordinate extreme values ​​of each corner point on the x-axis, y-axis, and z-axis are (x max , x min ), (y max ,y min ), (z max , z min ), obtain the coordinates of each point cloud on the x-axis, y-axis, and z-axis, and determine whether it is located at (x max , x min ), (y max ,y min ), (z max , z min ), if yes, it proves that the point cloud is a point cloud in the car and needs to be retained; if not, it means that it is a noise or background point cloud and needs to be removed to obtain the car point cloud data, reduce interference, and improve the processing efficiency of subsequent steps.

[0120] S4: segment the carriage point cloud data and perform plane fitting to obtain a number of planes and corresponding normal vectors, so as to obtain the material point cloud according to the normal vectors of each plane;

[0121] Specifically, it is optional but not limited to setting an angle threshold, dividing the car point cloud data into several point cloud clusters according to the specific structure of the car according to the car depth map and the car depth map, and performing plane fitting on each point cloud cluster to obtain several planes and corresponding normal vectors, obtain the angle between each normal vector and the z-axis, and determine whether it is greater than the angle threshold. If not, it remains unchanged. If so, all point clouds corresponding to the plane are eliminated, and the remaining point clouds are the material point clouds.

[0122] S41: segmenting the carriage point cloud data to obtain a number of point cloud clusters;

[0123] For example, the car point cloud can be optionally segmented into a number of point cloud clusters corresponding to the specific structures of the car according to the a priori car size and the specific structures of the car such as the car bottom and the car side rails.

[0124] S42: performing plane fitting on each point cloud cluster to obtain a number of planes and corresponding normal vectors;

[0125] Specifically, it is optional but not limited to setting the plane model according to the plane features and the equations that need to be satisfied. For example, the plane model is set with the plane normal vector and the constant term as parameters; more specifically, the plane model parameters can be optionally but not limited to be defined as: (A, B, C, D), where A, B, C are the components of the plane normal vector, D is the constant term of the plane equation, and the plane equation can be optionally but not limited to be expressed as: Ax+By+Cz+D=0; it is worth noting that the equations that need to be satisfied by the plane model and the parameters used are examples and are not limited to this.

[0126] Specifically, it is optional but not limited to randomly selecting 3 points P in each point cloud cluster a ,P b ,P c , is the current interior point, which is substituted into the plane equation to fit the parameters of the plane model. The plane normal vector calculation formula is: × is a cross multiplication operation. Substitute the normal vector and any point into the plane equation Ax+By+Cz+D=0 to obtain the equation coefficients of the plane equation corresponding to each point cloud cluster.

[0127] S43: Obtain the angle between each normal vector and the z-axis, and determine whether it is greater than the set angle threshold. If not, remain unchanged. If so, remove all point clouds corresponding to the plane to obtain the material point cloud.

[0128] Specifically, it is optional but not limited to setting an angle threshold, and then calculating the angle between the normal vector of the plane equation corresponding to each point cloud cluster and the z-axis, and judging whether it is greater than the set angle threshold, if not, it means that the plane is the plane corresponding to the material point cloud that needs to be retained, and if so, it means that the plane is the plane corresponding to the car side rail point cloud, and all point clouds corresponding to the plane need to be removed to obtain the material point cloud.

[0129] S5: Obtain the minimum external figure of each material point cloud to obtain the bounding box of each material point cloud and the coordinates of each bounding box according to the carriage depth map;

[0130] Specifically, it is optional but not limited to projecting the material point cloud onto the image plane, obtaining the minimum circumscribed figure of each material point cloud according to the circumscribed figure calculation method, and then combining the minimum circumscribed figure of each material point cloud with the height information corresponding to the material point cloud in the car depth map to obtain the bounding box of each material, and at the same time, obtain the coordinates of each bounding box in the acquisition device coordinate system according to the car depth map; the minimum circumscribed figure can optionally include any shape including rectangle, circle, ellipse, diamond, etc., which can be arbitrarily set by technical personnel in this field according to the specific shapes of the transportation equipment and materials.

[0131] Preferably, the coordinates of the bounding box in the acquisition device coordinate system can be optionally represented by the coordinates of the center point or the corner point.

[0132] S6: The coordinates of each bounding box are converted to the robot end coordinate system according to the conversion matrix to obtain the size and coordinates of each bounding box in the robot end coordinate system.

[0133] Specifically, the coordinates of each bounding box can be converted to the robotic arm end coordinate system according to the conversion matrix between the acquisition device coordinate system and the robotic arm end coordinate system in step S1, so as to obtain the size and coordinates of each bounding box in the robotic arm end coordinate system.

[0134] In this embodiment, a method for positioning and detecting materials in a carriage of the present invention is provided. By acquiring a conversion matrix between the acquisition device coordinate system and the mechanical arm end coordinate system, and then controlling the mechanical arm to move in a set direction, the carriage depth map is acquired, and the corner point coordinates and edge contours of the upper surface and the bottom of the carriage in the mechanical arm end coordinate system are obtained, so that initial point cloud data is generated according to the prior acquisition device internal parameters and the carriage depth map information, and the upper surface and the bottom of the carriage are segmented according to the corner points to obtain the carriage point cloud data, so as to complete the precise positioning of the carriage, segment the carriage point cloud data, and perform plane fitting to obtain a plurality of planes. and the corresponding normal vectors, so as to obtain the material point cloud according to the normal vectors of each plane, obtain the minimum external figure of each material point cloud, obtain the bounding box of each material point cloud and the coordinates of each bounding box according to the depth map of the carriage, convert the coordinates of each bounding box to the coordinate system of the end of the robot arm according to the transformation matrix, obtain the size and coordinates of each bounding box in the coordinate system of the end of the robot arm, generate an accurate model of the carriage by collecting the depth image and depth data of the carriage, and then extract the actual position and size information of the material from the inside of the carriage through point cloud processing and image processing technology, thereby enhancing the adaptability of the existing technology in different environments and the accuracy of material recognition. The existing technology solves the problems of insufficient adaptability and low accuracy in carriage positioning and size detection, as well as insufficient efficiency.

[0135] On the other hand, the present invention further provides a computer storage medium storing executable program code; the executable program code is used to execute any of the above-mentioned carriage positioning and material detection methods.

[0136] On the other hand, the present invention also provides a terminal device, including a memory and a processor; the memory stores program codes that can be executed by the processor; the program codes are used to execute any of the above-mentioned carriage positioning and material detection methods.

[0137] Exemplarily, the program code may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the program code in the terminal device.

[0138] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the terminal device may also include an input / output device, a network access device, a bus, etc.

[0139] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0140] The memory may be an internal storage unit of the terminal device, such as a hard disk or a memory. The memory may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device. Further, the memory may also include both an internal storage unit of the terminal device and an external storage device. The memory is used to store the program code and other programs and data required by the terminal device. The memory may also be used to temporarily store data that has been output or is to be output.

[0141] The above-mentioned computer storage medium and terminal device are created based on the above-mentioned carriage positioning and material detection method. Their technical functions and beneficial effects will not be repeated here. The various technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the various technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for carriage positioning and material detection, characterized in that: include: Obtain the transformation matrix between the acquisition device coordinate system and the robot end coordinate system; Control the robot arm to move in the set direction, collect the depth map of the car body, and obtain the coordinates of the corner points on the upper and bottom surfaces of the car body in the coordinate system of the robot arm end; Generate initial point cloud data based on the prior acquisition device internal parameters and the car depth map information, and segment the car according to the corner points of the upper surface and the bottom of the car to obtain the car point cloud data; The car body point cloud data is segmented and plane fitting is performed to obtain several planes and corresponding normal vectors, so as to obtain the material point cloud according to the normal vectors of each plane; Obtain the minimum external figure of each material point cloud to obtain the bounding box of each material point cloud and the coordinates of each bounding box according to the car depth map; The coordinates of each bounding box are converted to the robot end coordinate system according to the transformation matrix, and the size and coordinates of each bounding box in the robot end coordinate system are obtained.

2. The method according to claim 1, characterized in that The steps to obtain the transformation matrix include: S11: Install the calibration plate at the set position, control the acquisition device to scan the surface of the calibration plate, and obtain the coordinates and numbers corresponding to each feature point in the acquisition device coordinate system; S12: Install the calibration device at the end of the robot arm, and touch the feature points on the surface of the calibration plate in turn to obtain the coordinates and numbers corresponding to each feature point in the coordinate system of the end of the robot arm; S13: constructing a number of feature point pairs according to the numbers corresponding to the feature points, and establishing a linear equation system according to the feature point pairs; S14: Substitute the coordinates corresponding to each feature point into the linear equation group, and solve it to obtain the transformation matrix between the acquisition device coordinate system and the robot end coordinate system; S15: Set a number of verification points, and obtain the coordinates and numbers of each verification point in the acquisition device coordinate system and the robot end coordinate system; S16: Convert all verification points to the same coordinate system, obtain the distance between verification points with the same number, and determine whether it is less than the distance threshold. If so, the conversion matrix is ​​qualified. If not, return to step S11.

3. The method according to claim 1, characterized in that The steps of obtaining the coordinates of the corner points of the upper surface and the bottom of the carriage in the robot end coordinate system include: respectively collecting a carriage depth map of a first area of ​​the carriage and a carriage depth map of a second area of ​​the carriage; Perform gradient calculation on the depth map of each carriage to obtain the corresponding carriage contour image; The edge contour of the carriage is obtained according to the contour images of each carriage, so as to obtain the geometric data of the carriage according to the edge contour of the carriage and make a corresponding carriage template; Perform template matching on the carriage depth map according to each carriage template to obtain the coordinates of the corner points on the upper surface of the carriage, and map them to the bottom of the carriage according to the carriage depth map to obtain the coordinates of the corner points on the bottom of the carriage; According to the conversion matrix between the acquisition device coordinate system and the robot end coordinate system, the coordinates of the corner points on the upper surface and bottom of the carriage in the robot base coordinate system are obtained.

4. The method according to claim 3, characterized in that: The step of obtaining the coordinates of the corner points on the upper surface and the bottom of the carriage in the robot arm end coordinate system also includes: Dedistort the car depth map to obtain the corrected corner point coordinates; According to the prior carriage height, the carriage depth map is filtered to obtain an optimized carriage depth map.

5. The method according to claim 1, characterized in that The step of obtaining the coordinates of the corner points on the upper surface and the bottom of the carriage in the robot arm end coordinate system also includes: Scan the surface of the carriage at set intervals to obtain several sets of carriage depth maps; Perform gradient calculation on the depth map of each carriage to obtain the corresponding carriage contour image; A corresponding carriage template is made according to each carriage contour image, so as to perform template matching on the carriage depth map according to each carriage template to obtain the coordinates of the corner points on the upper surface of the carriage; According to the depth map of the car body, the coordinates of the corner points on the upper surface of the car body are mapped to the bottom of the car body to obtain the coordinates of the corner points on the bottom of the car body. According to the transformation matrix between the coordinate system of the acquisition device and the coordinate system of the end of the robotic arm, the coordinates of the corner points on the upper surface and bottom of the car body in the robot base coordinate system are obtained.

6. The method according to claim 5, characterized in that The steps of making corresponding carriage templates according to the carriage contour images, performing template matching on the carriage depth map according to the carriage templates, and obtaining the coordinates of the corner points on the upper surface of the carriage include: Obtaining a carriage edge contour according to each carriage contour image, so as to obtain carriage geometry data according to the carriage edge contour; According to the geometric data of the carriage, a corresponding carriage template is made, and the carriage depth map is template matched according to each carriage template to obtain the coordinates of the corner points of the front half of the carriage; According to the direction and distance between the prior midpoint of the carriage and the corner coordinates of the front half of the carriage, the corner coordinates of the rear of the carriage are reversely deduced to obtain the corner coordinates of the upper surface of the carriage.

7. The method according to claim 1, characterized in that The steps to obtain the car point cloud data include: According to the prior acquisition equipment internal parameters and the depth value of each pixel in the car depth map, the three-dimensional coordinates corresponding to each pixel are obtained to generate initial point cloud data; Get the three-dimensional coordinates of all corner points and get the extreme values ​​of the corner point coordinates; A filtering range is constructed according to the extreme values ​​of the corner point coordinates, so as to perform a filtering operation on the initial point cloud data according to the filtering range to obtain the carriage point cloud data.

8. The method according to claim 1, characterized in that The steps to obtain the material point cloud include: Segment the carriage point cloud data to obtain several point cloud clusters; Perform plane fitting on each point cloud cluster to obtain several planes and corresponding normal vectors; Get the angle between each normal vector and the z-axis to determine whether it is greater than the set angle threshold. If not, keep it unchanged. If so, remove all point clouds corresponding to the plane to obtain the material point cloud.

9. A computer storage medium, characterized in that An executable program code is stored; the executable program code is used to execute the carriage positioning and material detection method described in any one of claims 1-8.

10. A terminal device, characterized in that: It comprises a memory and a processor; the memory stores program codes executable by the processor; the program codes are used to execute the carriage positioning and material detection method described in any one of claims 1-8.

Images