Method, device and equipment for extracting region of interest of three-dimensional point cloud data and medium

Abstract

The invention relates to a region-of-interest extraction method and device for three-dimensional point cloud data, equipment and a medium, and the method comprises the steps: extracting target point cloud data of a target object from original three-dimensional point cloud data, and determining the coordinates of each original end point through a boundary straight line of the target object; after the original end point coordinates are converted into the camera coordinate system through the target rotation angle, second end point coordinates are determined again, all the second end point coordinates form a standard rectangular area, the standard area is the maximum rectangular area based on the camera coordinate system, and then the second end point coordinates are converted into the original coordinate system through the target rotation angle; according to the method, the three-dimensional ROI of the target object can be automatically extracted from the three-dimensional point cloud data of the industrial scene, and the extraction rate and the extraction precision are improved, and the maximum rectangular area is searched in the formed target quadrilateral area, and the three-dimensional ROI of the target object can be determined by combining the height of the target object.

Description

Technical Field

[0001] This application relates to the technical field of point cloud data processing, and particularly to a method, apparatus, device and medium for extracting a region of interest of three-dimensional point cloud data. Background Art

[0002] Currently, when extracting the 3D (Three-Dimensional) ROI (Region of Interest) of a target object in an industrial scenario, it is necessary to extract it from the three-dimensional point cloud data containing the target object in a completely manual operation manner. The specific process is as follows: The operator manually draws a bounding box or polygon in a professional point cloud processing software to select the target area in the three-dimensional point cloud scene. The target area contains the target object. For example, when the target object is an industrial belt, it is necessary to manually select the 3D ROI of the industrial belt from the three-dimensional point cloud data containing the industrial belt in order to perform an automated detection process on the industrial belt within the 3D ROI. However, this manual selection method requires the operator to have certain professional experience to accurately identify the position and range of the target object in the complex point cloud; moreover, this method requires the operator to repeatedly adjust the viewing angle and observe and confirm from multiple angles to complete a relatively accurate ROI selection. The entire process completely relies on manual judgment and operation without any automated auxiliary functions, resulting in problems such as low efficiency and poor stability. In addition, this manual method cannot be integrated into the automated detection process at all, seriously restricting the overall efficiency improvement of the intelligent manufacturing system.

[0003] Therefore, how to quickly and accurately extract the 3D ROI of a target object from three-dimensional point cloud data in an industrial scenario is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] This application provides a method, apparatus, device and medium for extracting a region of interest of three-dimensional point cloud data to quickly and accurately extract the 3D ROI of the three-dimensional point cloud data.

[0005] In a first aspect, this application provides a method for extracting a region of interest of three-dimensional point cloud data, including: Obtain the original three-dimensional point cloud data; Extract the target point cloud data of the target object from the original three-dimensional point cloud data; Determine the boundary lines of the target object on the target plane according to the target point cloud data, and determine the original endpoint coordinates according to each boundary line; Convert each original endpoint coordinate to a first endpoint coordinate using the target rotation angle, and determine the second endpoint coordinate according to the coordinate value of each first endpoint coordinate; wherein, the target rotation angle is the included angle between the boundary line and the corresponding coordinate axis direction vector, and the area formed by each second endpoint coordinate is a standard rectangular area; Convert each second endpoint coordinate to a target endpoint coordinate using the target rotation angle; Determine the target quadrilateral area according to each target endpoint coordinate, and determine the largest rectangular area within the target quadrilateral area; Determine the region of interest using the largest rectangular area and the height of the target object.

[0006] Optionally, determining the boundary line of the target object on the target plane according to the target point cloud data, and determining the original endpoint coordinate according to each boundary line includes: Extract the boundary point cloud data of the target object from the target point cloud data; Perform line fitting on the boundary point cloud data to determine the boundary line of the target object on the target plane; Take the coordinate of the intersection point of the straight lines between the boundary lines as the original endpoint coordinate.

[0007] Optionally, extracting the boundary point cloud data of the target object from the target point cloud data includes: Determine the normal vector of each data point in the target point cloud data; Calculate the included angle between the normal vectors of adjacent data points in the target point cloud data; Determine the boundary point cloud data according to the included angle between the normal vectors and the angle threshold.

[0008] Optionally, performing line fitting on the boundary point cloud data to determine the boundary line of the target object on the target plane includes: Determine each target line from the boundary point cloud data; the target line is determined by at least two target data points in the boundary point cloud data; Determine the boundary line that forms an acute angle with the target axis direction vector from each target line; Wherein, the target axis direction vector includes the horizontal axis direction vector and the vertical axis direction vector; the boundary lines include the upper boundary line, the lower boundary line, the left boundary line and the right boundary line of the target object on the target plane.

[0009] Optionally, the converting each original endpoint coordinate to a first endpoint coordinate using the target rotation angle includes: Determine the target boundary line from the boundary lines; the included angle between the target boundary line and the horizontal axis direction vector is an acute angle; Take the angle between the target boundary line and the horizontal axis direction vector as the target rotation angle; Determine the rotation matrix using the target rotation angle; Convert each original endpoint coordinate to the first endpoint coordinate through the rotation matrix.

[0010] Optionally, determining the second endpoint coordinates according to the coordinate values of each first endpoint coordinate includes: Determine the maximum value of the horizontal axis, the minimum value of the horizontal axis, the maximum value of the vertical axis, and the minimum value of the vertical axis according to each first endpoint coordinate; Determine the second endpoint coordinates according to the maximum value of the horizontal axis, the minimum value of the horizontal axis, the maximum value of the vertical axis, and the minimum value of the vertical axis; wherein, the vertical axis coordinate value of each second endpoint coordinate is the minimum vertical axis coordinate value of each first endpoint coordinate.

[0011] Optionally, determining the maximum rectangular area within the target quadrilateral area includes: Determine each current center point within the horizontal and vertical axis coordinate ranges of the target quadrilateral area; Determine candidate vertices according to each current center point; Judge whether each candidate vertex is located inside the target quadrilateral area; If so, determine the candidate rectangular area according to each candidate vertex; Take the candidate rectangular area with the largest area as the maximum rectangular area.

[0012] In a second aspect, the present application provides a device for extracting an area of interest from three-dimensional point cloud data, including: An acquisition module, configured to acquire original three-dimensional point cloud data; An extraction module, configured to extract the target point cloud data of the target object from the original three-dimensional point cloud data; A first determination module, configured to determine the boundary line of the target object on the target plane according to the target point cloud data, and determine the original endpoint coordinates according to each boundary line; A first conversion module, configured to convert each original endpoint coordinate to the first endpoint coordinate using the target rotation angle, and determine the second endpoint coordinates according to the coordinate values of each first endpoint coordinate; wherein, the target rotation angle is the angle between the boundary line and the corresponding coordinate axis direction vector, and the area formed by each second endpoint coordinate is a standard rectangular area; A second conversion module, configured to convert each second endpoint coordinate to the target endpoint coordinate using the target rotation angle; A second determination module, configured to determine the target quadrilateral area according to each target endpoint coordinate, and determine the maximum rectangular area within the target quadrilateral area; A third determination module, configured to determine a region of interest by using the maximum rectangular region and the height of the target object.

[0013] In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus; The memory is used to store a computer program; The processor is configured to implement the steps of the above-mentioned region of interest extraction method when executing the program stored on the memory.

[0014] In a fourth aspect, the present application further provides a computer storage medium, where the computer storage medium stores computer-executable instructions, and the computer-executable instructions are used to execute the steps of the above-mentioned region of interest extraction method of the present application.

[0015] The above technical solutions provided by the embodiments of the present application have the following advantages compared with the prior art: The present application discloses a method, device, equipment, and medium for extracting a region of interest from three-dimensional point cloud data. After extracting the target point cloud data of the target object from the original three-dimensional point cloud data, the original endpoint coordinates can be determined through the boundary straight line of the target object, and after converting the original endpoint coordinates to the camera coordinate system through the target rotation angle, the second endpoint coordinates are re-determined. The second endpoint coordinates form a standard rectangular region, and this standard region is the maximum rectangular region based on the camera coordinate system. Then, the second endpoint coordinates are converted to the original coordinate system through the target rotation angle, and the maximum rectangular region is searched within the formed target quadrilateral region, and combined with the height of the target object, the three-dimensional region of interest of the target object can be determined. Through this method, the present application can automatically extract the 3D ROI of the target object from the three-dimensional point cloud data of the industrial scene, improve the extraction rate and accuracy, provide reliable technical support for automated detection and intelligent manufacturing, and promote the transformation and upgrading of industrial production towards the intelligent direction. Description of the Drawings

[0016] The drawings here are incorporated into the specification and form a part of this specification, showing the embodiments consistent with the present invention and used together with the specification to explain the principles of the present invention.

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] One or more embodiments are exemplarily illustrated by the pictures in the corresponding drawings. These exemplary illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the drawings in the figures do not constitute a scale limitation.

[0019] Figure 1 Schematic flow chart of a method for extracting a region of interest from three-dimensional point cloud data provided by an embodiment of the present application; Figure 2 Schematic structural diagram of a device for extracting a region of interest from three-dimensional point cloud data provided by an embodiment of the present application; Figure 3 Schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0020] In the prior art, when extracting the ROI of three-dimensional point cloud data, operators need to extract it in a completely manual operation manner using professional point cloud processing software. The professional point cloud processing software can be CloudCompare (three-dimensional point cloud processing software), MeshLab (mesh processing system), etc. However, this completely manual ROI extraction method has many obvious defects: First of all, the operation efficiency is extremely low. Each ROI requires the operator to spend a lot of time manually selecting, which is difficult to meet the processing requirements of a large amount of data in industrial scenarios; Secondly, the extraction accuracy completely depends on the operator's experience and state. The ROIs extracted by different operators or even the same operator at different times will have significant differences, resulting in the lack of consistency and repeatability of the results; Furthermore, manual operations are prone to errors due to fatigue or inattention, and key areas may be missed or too much background noise may be included in complex scenarios; Finally, this manual method cannot be integrated into the automated detection process at all, which severely restricts the overall efficiency improvement of the intelligent manufacturing system.

[0021] Therefore, in order to solve the problems brought by the complete dependence on manual operations in the existing three-dimensional point cloud ROI extraction technology, the present application provides a method, device, equipment and medium for extracting a region of interest from three-dimensional point cloud data. This solution can realize the automatic analysis of the three-dimensional point cloud data of the target object and the accurate calculation of the maximum 3D ROI region. This solution breaks through the limitations of the traditional manual selection mode, adopts advanced point cloud processing algorithms to automatically identify the target area, and can quickly generate the optimal ROI without manual intervention, significantly improving the consistency of extraction efficiency and accuracy. At the same time, this solution has good adaptability and can meet the requirements of different industrial scenarios, providing reliable technical support for automated detection and intelligent manufacturing, and promoting the transformation and upgrading of industrial production towards the intelligent direction.

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are some but not all of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts fall within the scope of protection of this application.

[0023] The following disclosure provides many different embodiments or examples for implementing different structures of the present invention. To simplify the disclosure of the present invention, the components and settings of specific examples are described below. Of course, they are only examples and are not intended to limit the present invention. In addition, the present invention may repeat reference numerals and / or letters in different examples. This repetition is for the purpose of simplicity and clarity and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0024] See Figure 1 , which is a schematic flowchart of a method for extracting a region of interest from three-dimensional point cloud data provided by an embodiment of this application. The method includes: S101. Obtain the original three-dimensional point cloud data; In this application, the original three-dimensional point cloud data is data obtained by a structured light three-dimensional camera or the like. The original three-dimensional point cloud data includes the target object and the surrounding environment information. The target object is the object of interest that needs to be labeled by 3D ROI. The target object can be a belt or other objects, and is not specifically limited herein. In this embodiment, only the target object being the target belt is used as an example for illustration. If the target object is the target belt, then in this application, when using a structured light three-dimensional camera to obtain the original three-dimensional point cloud data of the industrial scene, it includes the three-dimensional point cloud on the surface of the target belt, as well as a large number of background objects and outlier noise points in the industrial scene, forming a complex three-dimensional scene.

[0025] S102. Extract the target point cloud data of the target object from the original three-dimensional point cloud data; Since the original three-dimensional point cloud data includes not only the point cloud data of the target object, but also the point cloud data of the background objects and outlier noise, this solution needs to extract the target point cloud data of the target object from the original three-dimensional point cloud data. This application can specifically apply a point cloud segmentation algorithm based on the Euclidean distance, and achieve the precise extraction of the target object point cloud by calculating the spatial distance matrix and setting a reasonable threshold.

[0026] S103. Determine the boundary lines of the target object on the target plane according to the target point cloud data, and determine the original endpoint coordinates according to each boundary line; In this application, after obtaining the target point cloud data by segmenting the point cloud of the target object, it is necessary to further extract the boundary lines of the target object. In this application, the boundary line is the boundary line of the target object on the target plane, and the number of boundary lines is related to the shape of the target plane. For example, if the target object is a cuboid with six planes and the top surface of the target object is used as the target plane, then the boundary lines are the four boundary lines of the top surface of the target object. Further, this application needs to calculate the pairwise intersections of the four boundary lines in sequence, solve the corresponding intersection coordinates, and these intersection coordinates are the original endpoint coordinates of the target object on the target plane. For example, if four boundary lines are recognized, since there are four intersections of the four boundary lines, the coordinates of the four intersections are used as the original endpoint coordinates.

[0027] S104. Use the target rotation angle to convert each original endpoint coordinate into a first endpoint coordinate, and determine the second endpoint coordinate according to the coordinate values of each first endpoint coordinate; wherein, the target rotation angle is the included angle between the boundary line and the corresponding coordinate axis direction vector, and the area formed by each second endpoint coordinate is a standard rectangular area; It should be noted that the original endpoint coordinates calculated in the above process are the coordinates in the target object coordinate system. To avoid the influence of lens distortion and perspective projection on the original endpoint coordinates, direct processing may lead to errors in the ROI. Therefore, when this application determines the ROI, the original endpoint coordinates can be converted from the target object coordinate system to the camera coordinate system, and after determining the ROI in the camera coordinate system, it is then converted back to the target object coordinate system.

[0028] When this application performs the conversion, it first needs to determine the target rotation angle, which is the included angle between one of the boundary lines and the corresponding coordinate axis direction vector; then construct a rotation matrix around the Z-axis (vertical axis) according to this target rotation angle, and this rotation matrix is used to transform each original endpoint coordinate into the corresponding first endpoint coordinate. For example, if the four original endpoint coordinates of the target belt are obtained, then this rotation matrix is applied for transformation to align the four original endpoint coordinates from the belt coordinate system to the camera coordinate system. After the rotation transformation, the four first endpoint coordinates of the target belt in the corrected coordinate system are obtained.

[0029] Further, after this application obtains the first endpoint coordinates of the target object in the camera coordinate system, it can re-determine the second endpoint coordinate according to the coordinate values of the first endpoint coordinates. The second endpoint coordinate can be determined by the maximum value of the horizontal axis (X-axis maximum value), the minimum value of the horizontal axis (X-axis minimum value), the maximum value of the vertical axis (Y-axis maximum value), and the minimum value of the vertical axis (Y-axis minimum value) of each first endpoint coordinate. Through the re-determined second endpoint coordinates, a new boundary range can be determined to form a standard rectangular area, and this standard rectangular area is the maximum region of interest of the target object in the camera coordinate system.

[0030] S105. Convert each second endpoint coordinate to a target endpoint coordinate using the target rotation angle; It should be noted that the second endpoint coordinates determined through the above process are coordinates in the camera coordinate system. In this application, it is necessary to convert each second endpoint coordinate to a target endpoint coordinate using the target rotation angle in order to restore from the camera coordinate system to the target object coordinate system.

[0031] S106. Determine a target quadrilateral region based on each target endpoint coordinate, and determine the largest rectangular region within the target quadrilateral region; After determining the converted target endpoint coordinates in this application, the target quadrilateral region can be determined based on the target endpoint coordinates. Since the region formed by the second endpoint coordinates determined in the camera coordinate system is a standard rectangular region, the target quadrilateral region determined by the converted target endpoint coordinates is specifically a parallelogram. Then, the largest rectangular region is determined within this target quadrilateral region, and this largest rectangular region is the largest rectangular region containing the target object.

[0032] S107. Determine the region of interest using the largest rectangular region and the height of the target object.

[0033] In this application, the largest rectangular region is the largest rectangular region of the target object on the target plane. For the Z-axis numerical range of the largest rectangular region, it can be determined according to the minimum Z value in this largest rectangular region and the height of the target object. For example, the Z-axis range is: minimum Z value ~ (minimum Z value + height of the target object). It can be seen that the region determined by the horizontal and vertical coordinate ranges of the above largest rectangular region and the Z-axis range in this application is the largest 3D ROI.

[0034] In summary, through the automated point cloud processing algorithm, this solution can quickly identify and accurately calculate the largest 3D ROI region, significantly improving the extraction efficiency and accuracy consistency, and avoiding the deficiencies of manual intervention; at the same time, this solution has good adaptability and can meet the requirements of different industrial scenarios, providing reliable support for automated detection and intelligent manufacturing, and promoting the intelligent transformation and upgrading of industrial production. Through this solution, the problems of low efficiency and unstable accuracy caused by relying on manual operations in the prior art are effectively solved.

[0035] In another embodiment of this application, the process of determining the boundary line of the target object on the target plane based on the target point cloud data and determining the original endpoint coordinates based on each boundary line specifically includes the following steps: Extract the boundary point cloud data of the target object from the target point cloud data; Perform linear fitting on the boundary point cloud data to determine the boundary line of the target object on the target plane; Use the coordinate of the intersection point of the lines between the boundary lines as the original endpoint coordinates.

[0036] It should be noted that when determining the original endpoint coordinates in this application, it is necessary to extract the boundary point cloud data of the target object from the target point cloud data. In another embodiment of this application, by determining the normal vectors of each data point in the target point cloud data, calculating the included angle between the normal vectors of adjacent data points in the target point cloud data, and determining the boundary point cloud data according to the included angle between the normal vectors and the angle threshold.

[0037] Specifically, the target point cloud data in this embodiment can be expressed as: , where the i-th data point has three-dimensional coordinates: . In order to accurately detect the boundary point cloud data in this application, it is necessary to first calculate the normal vector of each point. The estimation of the normal vector is usually based on the principal component analysis method, and is solved by constructing the local neighborhood of the point and calculating the covariance matrix.

[0038] For the point , its set of K nearest neighbor points is defined as: ; where represents the set of K nearest neighborhoods of the point , is the first K points sorted in ascending order of the distance from the point , that is, satisfying:

[0039] where represents the Euclidean distance between two points.

[0040] The covariance matrix is calculated as follows: ; where is the center of the local neighborhood of: ; Perform eigenvalue decomposition on the covariance matrix: ; where is the eigenvalue, is the corresponding eigenvector. The eigenvector corresponding to the smallest eigenvalue is the normal vector of this point.

[0041] After obtaining the normal vectors of all data points, calculate the included angle between the normal vectors of adjacent data points to judge the boundary information of the point cloud. For the point and its neighbor point , the included angle of the normal vectors is calculated by the following formula: .

[0042] where represents the normal vector of, represents The normal vector of represents the dot product of the normal vectors, and represents the product of the magnitudes of the normal vectors. If is greater than the set angle threshold , then this point may belong to the boundary point.

[0043] In this application, in order to further improve the robustness of detection, a statistical method is used to determine whether the point is a boundary point. Define the boundary feature score of the point as:

[0044] where is an indicator function. If , it returns 1, otherwise it returns 0. If exceeds the set threshold, then the point is considered a boundary point; that is: select each neighbor point from the set of K-nearest neighbor points of the data point . If the total number of the included angles between the normal vectors of the data point and the neighbor points that exceed the angle threshold exceeds the set threshold, then it is determined that the data point is a boundary point.

[0045] Through the above process, this application can determine the boundary point cloud data from the target point cloud data. In another embodiment of this application, the process of performing a straight line fitting on the boundary point cloud data to determine the boundary straight line of the target object on the target plane includes: determining each target straight line from the boundary point cloud data; the target straight line is determined by at least two target data points in the boundary point cloud data; determining the boundary straight line that forms an acute angle with the target axis direction vector from each target straight line; where the target axis direction vector includes the horizontal axis direction vector and the vertical axis direction vector; the boundary straight lines include the upper boundary straight line, the lower boundary straight line, the left boundary straight line, and the right boundary straight line of the target object on the target plane.

[0046] Specifically, after this application extracts the boundary point cloud data of the target object, it is necessary to perform a straight line fitting on the boundary points and sequentially extract the upper, lower, left, and right four boundary straight lines of the target object. Therefore, when performing a straight line fitting in this application, it is necessary to sequentially fit two straight lines that form an acute angle with the horizontal axis direction vector and two straight lines that form an acute angle with the vertical axis direction vector. Here, the specific steps of performing a straight line fitting on this application are described: a. Randomly sample the boundary point cloud data, set the threshold to 1, and obtain the best straight line. The following are the steps for fitting the first straight line that forms an acute angle with the direction vector (1, 0, 0): Randomly select the minimum number of points from the data; among them, when performing a straight line fitting in this application, at least 2 target data points are selected.

[0047] Calculate a model through at least two target data points, where the model is a target straight line; and calculate whether the angle between the target straight line and the direction vector (1, 0, 0) (the horizontal axis direction vector) is an acute angle, and retain the target straight lines for which the angle is acute.

[0048] Then calculate the distances from all data points to this model (the target straight line), and count the points that meet the distance threshold; in this application, specifically count the data points whose distance from the target straight line is less than 1, and call this data point an inlier.

[0049] For each target straight line in this application, it is necessary to calculate the inliers of each target straight line through the above process. If the number of inliers of the current target straight line is more than that of the previous best model, then use the current target straight line as the best model, repeat the above steps several times, and finally select the best model with the largest number of inliers as the best fit, that is, the boundary straight line in this application.

[0050] b. After obtaining the first boundary straight line through step a, it is necessary to calculate the distance from each point to this straight line : ; where is a point on the straight line, and d is the unit direction vector of the straight line. If is less than the set threshold, then it is regarded as an inlier and this point needs to be removed.

[0051] c. After removing the point cloud on and near the fitted boundary straight line, the remaining point cloud is again fitted in the manner of steps a - b. After the fitting is completed, generate another boundary straight line whose angle with the direction vector (1, 0, 0) is an acute angle, and remove the point cloud on and near the fitted boundary straight line to obtain the filtered point cloud data.

[0052] d. For the filtered point cloud data, continue to apply the random sampling algorithm, but this time the screening target is a straight line whose angle with the direction vector (0, 1, 0) (the vertical axis direction vector) is an acute angle, and obtain two boundary straight lines whose angles with the direction vector (0, 1, 0) are acute angles.

[0053] e. Through the above steps, the equations of four boundary straight lines can finally be obtained.

[0054] As can be seen from the above, when determining the original endpoint coordinates in this application, the boundary point cloud data can be specifically determined according to the normal vector angles and the angle threshold of each data point in the target point cloud data. In this boundary point cloud data, by means of linear fitting, two lines that are acute angles with the horizontal axis direction vector and two lines that are acute angles with the vertical axis direction vector are fitted. The above four fitted lines are used as the boundary lines of the target object. Through this method, the boundary lines of the target object can be quickly and accurately found in this application. Furthermore, the original endpoint coordinates can be determined through the intersection coordinates of the boundary lines.

[0055] In another embodiment of this application, the process of converting each original endpoint coordinate into a first endpoint coordinate by using the target rotation angle includes: Determine the target boundary line from the boundary lines; the angle between the target boundary line and the horizontal axis direction vector is an acute angle; use the angle between the target boundary line and the horizontal axis direction vector as the target rotation angle; determine the rotation matrix by using the target rotation angle, and convert each original endpoint coordinate into a first endpoint coordinate through the rotation matrix.

[0056] In this application, four boundary lines are obtained through the above fitting process. When determining the target rotation angle in this application, specifically, find the target boundary line with an acute angle with the horizontal axis direction vector (X-axis direction vector (1, 0, 0)) from the four boundary lines. In this embodiment, the first boundary line determined by random sampling in the above embodiment can be used as the target boundary line. Then calculate the angle between the target boundary line and the horizontal axis direction vector. This application refers to this angle as the target rotation angle, and this target rotation angle reflects the offset angle of the target object coordinate system relative to the camera coordinate system. After determining the rotation matrix through the target rotation angle in this application, each original endpoint coordinate can be converted into a first endpoint coordinate through the rotation matrix.

[0057] In another embodiment of this application, after converting each original endpoint coordinate into a first endpoint coordinate through the rotation matrix, it is also necessary to determine the maximum value of the horizontal axis, the minimum value of the horizontal axis, the maximum value of the vertical axis, and the minimum value of the vertical axis according to each first endpoint coordinate; determine the second endpoint coordinates according to the maximum value of the horizontal axis, the minimum value of the horizontal axis, the maximum value of the vertical axis, and the minimum value of the vertical axis; among them, the vertical axis coordinate value of each second endpoint coordinate is the minimum vertical axis coordinate value of each first endpoint coordinate.

[0058] That is to say, after the present application converts each original endpoint coordinate into the camera coordinate system through the rotation matrix, the maximum rectangle can be determined in the camera coordinate system; in this embodiment, it is necessary to determine the minimum X value (the minimum value on the horizontal axis), the maximum X value (the maximum value on the horizontal axis), the minimum Y value (the minimum value on the vertical axis), and the maximum Y value (the maximum value on the vertical axis) according to each first endpoint coordinate, and construct four new endpoints through the above values to form a standard rectangle area, and uniformly set the Z coordinate value (the vertical axis coordinate value) of all endpoints to the minimum Z value (the minimum vertical axis coordinate value) among the four endpoints to ensure that the standard rectangle area is on the same plane. At this time, this standard rectangle area is the maximum 3D ROI after the target object coordinate system is converted to the camera coordinate system.

[0059] After the present application determines each second endpoint coordinate, it is also necessary to reverse-rotate each second endpoint coordinate through the rotation matrix determined by the target rotation angle to restore it from the camera coordinate system to the target object coordinate system, so as to obtain the maximum parallelogram area of the target object.

[0060] In summary, in the present application, in order to obtain the most accurate maximum rectangle area of the target object, the original endpoint coordinates can be converted from the target object coordinate system to the camera coordinate system according to the target rotation angle, and the maximum rectangle area of the target object in the camera coordinate system can be determined through the coordinate values of each endpoint. And the maximum rectangle area can be made to be on the same plane by unifying the vertical axis coordinate values, and then the endpoint coordinates of the standard rectangle area are reverse-rotated through the target rotation angle to restore it from the camera coordinate system to the target object coordinate system, so as to obtain the largest parallelogram area on the target object.

[0061] In another embodiment of the present application, the process of determining the maximum rectangle area within the target quadrilateral area specifically includes the following content: within the horizontal and vertical axis coordinate ranges of the target quadrilateral area, determine each current center point; determine candidate vertices according to each current center point; determine whether each candidate vertex is located inside the target quadrilateral area; if so, determine a candidate rectangle area according to each candidate vertex; and use the candidate rectangle area with the largest area as the maximum rectangle area.

[0062] In this embodiment, after determining the target quadrilateral area of the target object through the above embodiment, the X-axis coordinate value range and the Y-axis coordinate value range of the target quadrilateral area can be calculated, and the maximum and minimum values of the X-axis coordinate value range and the Y-axis coordinate value range are rounded to ensure that the area boundary is aligned to integer coordinates, which is convenient for subsequent processing and calculation.

[0063] The present application needs to find the maximum rectangle area that can be completely contained within the target quadrilateral area. The specific search process includes the following steps: a. Traversal search: Traverse within the X-axis coordinate value range and Y-axis coordinate value range of the parallelogram to determine the current point, and check possible rectangular regions point by point with an integer step size.

[0064] b. Construct four candidate vertices of the rectangle: For the current point , calculate the corresponding four candidate vertices based on central symmetry to form a candidate rectangle.

[0065] c. Calculate the cross product: For each side of the quadrilateral (for example, side ), calculate the cross product from any candidate vertex to each side. In this application, any candidate vertex is represented by point , and the cross product formula is as follows: ; where, is the x coordinate of point A, is the x coordinate of point B, is the x coordinate of point P, is the y coordinate of point A, is the y coordinate of point B, is the y coordinate of point P.

[0066] d. Check the cross product sign: Calculate the cross products of each candidate vertex with the four sides of the target quadrilateral and check the sign of each cross product. If the four cross products of a certain candidate vertex are all greater than or equal to 0, or all less than or equal to 0, it means that the candidate vertex is inside the target quadrilateral. Otherwise, the candidate vertex is outside the target quadrilateral; in this application, select the four candidate vertices that are all inside the target quadrilateral to form a candidate rectangular region.

[0067] e. Calculate the rectangle area: For the candidate rectangular region that meets the conditions, calculate its area and compare it with the current maximum area to update the position information of the maximum candidate rectangular region.

[0068] f. Final result: After the traversal is completed, the candidate rectangular region with the largest area found is the largest rectangular region inside the target quadrilateral.

[0069] Through the above process, this application can determine the largest rectangular region of the target object on the X-axis and Y-axis, and then determine the Z-axis range of the largest rectangular region by the minimum Z value in the largest rectangular region and the height of the target object. The finally calculated region is the largest 3D ROI.

[0070] In summary, the present application proposes an intelligent maximum 3D ROI extraction solution based on 3D vision. This solution analyzes and processes the point cloud of the target object, and accurately calculates the maximum enclosed 3D ROI area on the premise that the XY coordinate axes of the ROI are parallel to the XY axes of the camera coordinate system. This method calculates completely relying on the belt point cloud data without manual intervention, can efficiently extract the maximum three-dimensional ROI of the workpiece area, improve the extraction accuracy and efficiency, and make it more suitable for automated inspection and intelligent manufacturing.

[0071] See Figure 2 , Figure 2 which is a schematic structural diagram of a device for extracting an interested region of three-dimensional point cloud data provided by an embodiment of the present application. The device specifically includes: An acquisition module 11, configured to acquire original three-dimensional point cloud data; An extraction module 12, configured to extract target point cloud data of a target object from the original three-dimensional point cloud data; A first determination module 13, configured to determine boundary lines of the target object on a target plane according to the target point cloud data, and determine original endpoint coordinates according to the boundary lines; A first conversion module 14, configured to convert each original endpoint coordinate into a first endpoint coordinate by using a target rotation angle, and determine a second endpoint coordinate according to the coordinate values of each first endpoint coordinate; wherein, the target rotation angle is the included angle between the boundary line and the corresponding coordinate axis direction vector, and the area formed by each second endpoint coordinate is a standard rectangular area; A second conversion module 15, configured to convert each second endpoint coordinate into a target endpoint coordinate by using the target rotation angle; A second determination module 16, configured to determine a target quadrilateral area according to each target endpoint coordinate, and determine a maximum rectangular area within the target quadrilateral area; A third determination module 17, configured to determine an interested region by using the maximum rectangular area and the height of the target object.

[0072] As an optional embodiment, the first determination module includes: An extraction unit, configured to extract boundary point cloud data of the target object from the target point cloud data; A fitting unit, configured to perform straight line fitting on the boundary point cloud data to determine the boundary lines of the target object on the target plane; A coordinate determination unit, configured to use the straight line intersection point coordinates between the boundary lines as the original endpoint coordinates.

[0073] As an alternative embodiment, the extraction unit is specifically configured to: determine the normal vectors of the data points in the target point cloud data, calculate the included angles between the normal vectors of adjacent data points in the target point cloud data, and determine the boundary point cloud data according to the included angles between the normal vectors and the angle threshold.

[0074] As an alternative embodiment, the fitting unit is specifically configured to: determine each target line from the boundary point cloud data; the target line is determined by at least two target data points in the boundary point cloud data; determine the boundary lines that form acute angles with the target axis direction vectors from the target lines; wherein the target axis direction vectors include the horizontal axis direction vector and the vertical axis direction vector; the boundary lines include the upper boundary line, the lower boundary line, the left boundary line, and the right boundary line of the target object on the target plane.

[0075] As an alternative embodiment, the first conversion module includes: A line determination unit, configured to determine a target boundary line from the boundary lines; the included angle between the target boundary line and the horizontal axis direction vector is an acute angle; An angle determination unit, configured to use the included angle between the target boundary line and the horizontal axis direction vector as the target rotation angle; A matrix determination unit, configured to determine a rotation matrix using the target rotation angle; A conversion unit, configured to convert each original endpoint coordinate into a first endpoint coordinate through the rotation matrix.

[0076] As an alternative embodiment, the first conversion module further includes: A coordinate value determination unit, configured to determine the maximum value of the horizontal axis, the minimum value of the horizontal axis, the maximum value of the vertical axis, and the minimum value of the vertical axis according to each first endpoint coordinate; An endpoint determination unit, configured to determine a second endpoint coordinate according to the maximum value of the horizontal axis, the minimum value of the horizontal axis, the maximum value of the vertical axis, and the minimum value of the vertical axis; wherein the vertical axis coordinate value of each second endpoint coordinate is the minimum vertical axis coordinate value of each first endpoint coordinate.

[0077] As an alternative embodiment, the second determination module includes: A center point determination unit, configured to determine each current center point within the horizontal and vertical axis coordinate ranges of the target quadrilateral region; A candidate vertex determination unit, configured to determine candidate vertices according to each current center point; A judgment unit, configured to judge whether each candidate vertex is located inside the target quadrilateral region; if so, determine a candidate rectangular region according to each candidate vertex; A region determination unit, configured to use the candidate rectangular region with the largest area as the largest rectangular region.

[0078] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0079] See Figure 3 , Figure 3 which is a schematic structural diagram of an electronic device provided by an embodiment of the present application, including a processor 21, a communication interface 22, a memory 23, and a communication bus 24. Among them, the processor 21, the communication interface 22, and the memory 23 communicate with each other through the communication bus 24; The memory 23 is used to store a computer program; When the processor 21 is used to execute the program stored on the memory 23, it implements the steps of the region of interest extraction method described in any of the above method embodiments, which will not be elaborated herein.

[0080] The communication bus mentioned in the above terminal may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity of representation, Figure 3 only a thick line is shown in [reference], but it does not mean that there is only one bus or one type of bus.

[0081] The communication interface is used for communication between the above terminal and other devices.

[0082] The memory may include a Random Access Memory (RAM), or may also include a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0083] The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP for short), an application specific integrated circuit (ASIC for short), a field-programmable gate array (FPGA for short), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0084] In another exemplary embodiment, a computer storage medium is also provided. When the program instructions are executed by a processor, the steps of the region of interest extraction method described in any of the above method embodiments are implemented. Among them, the storage medium may include: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc.

[0085] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and will not be repeated here.

[0086] It should be understood that the terms used herein are for the purpose of describing specific example embodiments only and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" as used herein may also include the plural forms. The terms "include", "comprise", "contain", and "have" are inclusive and thus specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or their combinations. The method steps, processes, and operations described herein are not to be construed as necessarily requiring them to be performed in the particular order described or illustrated, unless the order of execution is explicitly stated. It should also be understood that additional or alternative steps may be used.

[0087] The above description is only the specific implementation manners of the present invention, enabling those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for extracting a region of interest from three-dimensional point cloud data, characterized in that: include: Obtain original 3D point cloud data; Extracting target point cloud data of a target object from the original three-dimensional point cloud data; Determine the boundary lines of the target object on the target plane according to the target point cloud data, and determine the original endpoint coordinates according to each boundary line; The target rotation angle is used to convert each original endpoint coordinate into a first endpoint coordinate, and the second endpoint coordinate is determined according to the coordinate value of each first endpoint coordinate; wherein the target rotation angle is the angle between the boundary line and the corresponding coordinate axis direction vector, and the area formed by each second endpoint coordinate is a standard rectangular area; Using the target rotation angle, converting each second endpoint coordinate into a target endpoint coordinate; Determine a target quadrilateral region according to the coordinates of each target endpoint, and determine a maximum rectangular region within the target quadrilateral region; The region of interest is determined using the maximum rectangular region and the height of the target object.

2. The method for extracting a region of interest according to claim 1, characterized in that: Determining the boundary lines of the target object on the target plane according to the target point cloud data, and determining the original endpoint coordinates according to each boundary line includes: Extracting boundary point cloud data of the target object from the target point cloud data; Performing straight line fitting on the boundary point cloud data to determine the boundary straight line of the target object on the target plane; The coordinates of the intersection of the boundary lines are used as the original endpoint coordinates.

3. The method for extracting a region of interest according to claim 2, characterized in that: Extracting boundary point cloud data of the target object from the target point cloud data includes: Determine the normal vector of each data point in the target point cloud data; Calculating the normal vector angles of adjacent data points in the target point cloud data; The boundary point cloud data is determined according to the normal vector angle and the angle threshold.

4. The method for extracting a region of interest according to claim 2, characterized in that: Performing straight line fitting on the boundary point cloud data to determine the boundary straight line of the target object on the target plane includes: Determine each target straight line from the boundary point cloud data; the target straight line is determined by at least two target data points in the boundary point cloud data; Determine a boundary line which forms an acute angle with the target axis direction vector from each target line; The target axis direction vector includes a horizontal axis direction vector and a vertical axis direction vector; the boundary lines include an upper boundary line, a lower boundary line, a left boundary line and a right boundary line of the target object in the target plane.

5. The method for extracting a region of interest according to claim 1, characterized in that: The method of converting each original endpoint coordinate into the first endpoint coordinate by using the target rotation angle includes: Determine a target boundary line from the boundary lines; the angle between the target boundary line and the horizontal axis direction vector is an acute angle; The angle between the target boundary line and the horizontal axis direction vector is used as the target rotation angle; Determining a rotation matrix using the target rotation angle; Each original endpoint coordinate is converted into the first endpoint coordinate by the rotation matrix.

6. The method for extracting a region of interest according to claim 1, characterized in that: Determining the second endpoint coordinates according to the coordinate values ​​of the first endpoint coordinates includes: Determine the maximum value of the horizontal axis, the minimum value of the horizontal axis, the maximum value of the vertical axis, and the minimum value of the vertical axis according to the coordinates of each first endpoint; The second endpoint coordinates are determined according to the horizontal axis maximum value, the horizontal axis minimum value, the vertical axis maximum value and the vertical axis minimum value; wherein the vertical axis coordinate value of each second endpoint coordinate is the minimum vertical axis coordinate value of each first endpoint coordinate.

7. The method for extracting a region of interest according to any one of claims 1 to 6, characterized in that: Determining a maximum rectangular area within the target quadrilateral area includes: Determine each current center point within the horizontal and vertical axis coordinate range of the target quadrilateral area; Determine candidate vertices based on each current center point; Determine whether all candidate vertices are located inside the target quadrilateral area; If yes, determine the candidate rectangular area based on each candidate vertex; The candidate rectangular region with the largest area is taken as the largest rectangular region.

8. A device for extracting regions of interest from three-dimensional point cloud data, characterized in that: include: An acquisition module is used to acquire original three-dimensional point cloud data; An extraction module, used to extract target point cloud data of a target object from the original three-dimensional point cloud data; A first determination module is used to determine the boundary lines of the target object on the target plane according to the target point cloud data, and determine the original endpoint coordinates according to each boundary line; A first conversion module, used to convert each original endpoint coordinate into a first endpoint coordinate using a target rotation angle, and determine the second endpoint coordinate according to the coordinate value of each first endpoint coordinate; wherein the target rotation angle is the angle between the boundary straight line and the corresponding coordinate axis direction vector, and the area formed by each second endpoint coordinate is a standard rectangular area; A second conversion module, used to convert each second endpoint coordinate into a target endpoint coordinate using the target rotation angle; A second determination module is used to determine a target quadrilateral region according to the coordinates of each target endpoint, and determine a maximum rectangular region within the target quadrilateral region; The third determination module is used to determine the region of interest by using the maximum rectangular area and the height of the target object.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor is used to implement the steps of the region of interest extraction method described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, and the computer executable instructions are used to execute the steps of the region of interest extraction method described in any one of claims 1 to 7 of the present application.

Images