Method, apparatus, device and medium for extracting region of interest from three-dimensional point cloud data

Abstract

The present application relates to a method, apparatus, device and medium for extracting a region of interest from three-dimensional point cloud data. After extracting the target point cloud data of the target object from the original three-dimensional point cloud data, the original endpoint coordinates can be determined through the boundary straight line of the target object, and after converting the original endpoint coordinates to the camera coordinate system through the target rotation angle, the second endpoint coordinates are re-determined. The second endpoint coordinates form a standard rectangular area, and the standard area is the largest rectangular area based on the camera coordinate system. Then, the second endpoint coordinates are converted to the original coordinate system through the target rotation angle, and the largest rectangular area is searched within the formed target quadrilateral area, and the three-dimensional region of interest of the target object can be determined by combining the height of the target object. Through this method, the present application can automatically extract the 3D ROI of the target object from the three-dimensional point cloud data of the industrial scene, improving the extraction rate and accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of point cloud data processing, and in particular, to a method, device, equipment and medium for extracting a region of interest of three-dimensional point cloud data. Background Technique

[0002] Currently, when extracting the 3D (Three-Dimensional) ROI (Region of Interest) of a target object in an industrial scenario, it is necessary to extract it from the three-dimensional point cloud data containing the target object in a completely manual operation mode. The specific process is as follows: The operator manually draws a bounding box or polygon in a professional point cloud processing software to select the target area in the three-dimensional point cloud scene. The target area contains the target object. For example, when the target object is an industrial belt, it is necessary to manually select the 3D ROI of the industrial belt from the three-dimensional point cloud data containing the industrial belt in order to perform an automated detection process on the industrial belt within the 3D ROI. However, this manual selection method requires the operator to have certain professional experience to accurately identify the position and range of the target object in the complex point cloud; moreover, this method requires the operator to repeatedly adjust the viewing angle and observe and confirm from multiple angles to complete a relatively accurate ROI selection. The entire process completely relies on manual judgment and operation without any automated auxiliary functions, and there are problems such as low efficiency and poor stability. In addition, this manual method cannot be integrated into the automated detection process at all, which severely restricts the overall efficiency improvement of the intelligent manufacturing system.

[0003] Therefore, how to quickly and accurately extract the 3D ROI of a target object from three-dimensional point cloud data in an industrial scenario is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] The present application provides a method, device, equipment and medium for extracting a region of interest of three-dimensional point cloud data to quickly and accurately extract the 3D ROI of the three-dimensional point cloud data.

[0005] In a first aspect, the present application provides a method for extracting a region of interest of three-dimensional point cloud data, including:

[0006] Obtain the original three-dimensional point cloud data;

[0007] Extract the target point cloud data of the target object from the original three-dimensional point cloud data;

[0008] Determine the boundary lines of the target object on the target plane according to the target point cloud data, and determine the original endpoint coordinates according to each boundary line;

[0009] Convert each original endpoint coordinate to a first endpoint coordinate using the target rotation angle, and determine the second endpoint coordinate according to the coordinate values of each first endpoint coordinate; wherein, the target rotation angle is the included angle between the boundary line and the corresponding coordinate axis direction vector, and the area formed by each second endpoint coordinate is a standard rectangular area;

[0010] Use the target rotation angle to convert each second endpoint coordinate to a target endpoint coordinate;

[0011] Determine the target quadrilateral area according to each target endpoint coordinate, and determine the largest rectangular area within the target quadrilateral area;

[0012] Use the largest rectangular area and the height of the target object to determine the region of interest.

[0013] Optionally, according to the target point cloud data, determine the boundary line of the target object on the target plane, and determine the original endpoint coordinate according to each boundary line, including:

[0014] Extract the boundary point cloud data of the target object from the target point cloud data;

[0015] Perform line fitting on the boundary point cloud data to determine the boundary line of the target object on the target plane;

[0016] Take the coordinate of the intersection point of the lines between the boundary lines as the original endpoint coordinate.

[0017] Optionally, extracting the boundary point cloud data of the target object from the target point cloud data includes:

[0018] Determine the normal vector of each data point in the target point cloud data;

[0019] Calculate the included angle between the normal vectors of adjacent data points in the target point cloud data;

[0020] Determine the boundary point cloud data according to the included angle between the normal vectors and the angle threshold.

[0021] Optionally, performing line fitting on the boundary point cloud data to determine the boundary line of the target object on the target plane includes:

[0022] Determine each target line from the boundary point cloud data; the target line is determined by at least two target data points in the boundary point cloud data;

[0023] Determine the boundary line that forms an acute angle with the target axis direction vector from each target line;

[0024] Among them, the target axis direction vector includes a horizontal axis direction vector and a vertical axis direction vector; the boundary lines include the upper boundary line, the lower boundary line, the left boundary line, and the right boundary line of the target object in the target plane.

[0025] Optionally, the converting each original endpoint coordinate into a first endpoint coordinate by using the target rotation angle includes:

[0026] Determine a target boundary line from the boundary lines; the included angle between the target boundary line and the horizontal axis direction vector is an acute angle;

[0027] Take the included angle between the target boundary line and the horizontal axis direction vector as the target rotation angle;

[0028] Determine a rotation matrix by using the target rotation angle;

[0029] Convert each original endpoint coordinate into a first endpoint coordinate through the rotation matrix.

[0030] Optionally, determining a second endpoint coordinate according to the coordinate values of each first endpoint coordinate includes:

[0031] Determine the maximum value of the horizontal axis, the minimum value of the horizontal axis, the maximum value of the vertical axis, and the minimum value of the vertical axis according to each first endpoint coordinate;

[0032] Determine the second endpoint coordinate according to the maximum value of the horizontal axis, the minimum value of the horizontal axis, the maximum value of the vertical axis, and the minimum value of the vertical axis; wherein, the vertical axis coordinate value of each second endpoint coordinate is the minimum vertical axis coordinate value of each first endpoint coordinate.

[0033] Optionally, determining the maximum rectangular area within the target quadrilateral area includes:

[0034] Determine each current center point within the horizontal and vertical axis coordinate ranges of the target quadrilateral area;

[0035] Determine candidate vertices according to each current center point;

[0036] Judge whether each candidate vertex is located inside the target quadrilateral area;

[0037] If so, determine a candidate rectangular area according to each candidate vertex;

[0038] Take the candidate rectangular area with the largest area as the maximum rectangular area.

[0039] In a second aspect, the present application provides a device for extracting an interested area of three-dimensional point cloud data, including:

[0040] An acquisition module, configured to acquire original three-dimensional point cloud data;

[0041] An extraction module for extracting target point cloud data of a target object from the original three-dimensional point cloud data;

[0042] A first determination module for determining boundary lines of the target object on a target plane according to the target point cloud data, and determining original endpoint coordinates according to the boundary lines;

[0043] A first conversion module for converting each original endpoint coordinate into a first endpoint coordinate by using a target rotation angle, and determining a second endpoint coordinate according to the coordinate values of each first endpoint coordinate; wherein, the target rotation angle is the angle between the boundary line and the corresponding coordinate axis direction vector, and the area formed by each second endpoint coordinate is a standard rectangular area;

[0044] A second conversion module for converting each second endpoint coordinate into a target endpoint coordinate by using the target rotation angle;

[0045] A second determination module for determining a target quadrilateral area according to each target endpoint coordinate, and determining a maximum rectangular area within the target quadrilateral area;

[0046] A third determination module for determining a region of interest by using the maximum rectangular area and the height of the target object.

[0047] In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus;

[0048] The memory is used for storing a computer program;

[0049] The processor, when executing the program stored on the memory, implements the steps of the above-mentioned region of interest extraction method.

[0050] In a fourth aspect, the present application further provides a computer storage medium, and the computer storage medium stores computer-executable instructions for executing the steps of the above-mentioned region of interest extraction method of the present application.

[0051] The above technical solution provided by the embodiments of the present application has the following advantages compared with the prior art: The present application discloses a method, device, equipment and medium for extracting regions of interest from three-dimensional point cloud data. After extracting the target point cloud data of the target object from the original three-dimensional point cloud data, the original endpoint coordinates can be determined through the boundary lines of the target object, and after converting the original endpoint coordinates to the camera coordinate system through the target rotation angle, the second endpoint coordinates are re-determined. The second endpoint coordinates form a standard rectangular area, and this standard area is the largest rectangular area based on the camera coordinate system. Then, the second endpoint coordinates are converted back to the original coordinate system through the target rotation angle, and the largest rectangular area is searched within the formed target quadrilateral area, and combined with the height of the target object, the three-dimensional region of interest of the target object can be determined. Through this method, the present application can automatically extract the 3D ROI of the target object from the three-dimensional point cloud data of the industrial scene, improve the extraction rate and accuracy, provide reliable technical support for automated detection and intelligent manufacturing, and promote the transformation and upgrading of industrial production towards the intelligent direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present invention and used together with the specification to explain the principles of the present invention.

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0054] One or more embodiments are exemplarily illustrated by the pictures in the corresponding accompanying drawings. These exemplary illustrations do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements, unless otherwise stated, and the drawings in the drawings do not constitute a proportional limitation.

[0055] Figure 1 Schematic flow chart of a method for extracting regions of interest from three-dimensional point cloud data provided by the embodiments of the present application;

[0056] Figure 2 Schematic structural diagram of a device for extracting regions of interest from three-dimensional point cloud data provided by the embodiments of the present application;

[0057] Figure 3 Schematic structural diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0058] In the prior art, when extracting the ROI of 3D point cloud data, operators need to extract it in a completely manual operation mode using professional point cloud processing software, such as CloudCompare (3D point cloud processing software), MeshLab (mesh processing system), etc. However, this completely manual ROI extraction method has many obvious defects:

[0059] First of all, the operation efficiency is extremely low. For each ROI, the operator needs to spend a lot of time manually bounding, which is difficult to meet the processing requirements of a large amount of data in industrial scenarios;

[0060] Secondly, the extraction accuracy completely depends on the operator's experience and state. There will be significant differences in the ROIs extracted by different operators or even the same operator at different times, resulting in a lack of consistency and repeatability of the results;

[0061] Moreover, manual operations are prone to errors due to fatigue or inattention, and key areas may be missed or excessive background noise may be included in complex scenarios;

[0062] Finally, this manual method cannot be integrated into the automated detection process at all, which severely restricts the overall efficiency improvement of the intelligent manufacturing system.

[0063] Therefore, in order to solve the problems brought by the complete dependence on manual operations in the existing 3D point cloud ROI extraction technology, this application provides a method, device, equipment and medium for extracting the region of interest of 3D point cloud data. This solution can realize the automatic analysis of the 3D point cloud data of the target object and the accurate calculation of the maximum 3D ROI area. This solution breaks through the limitations of the traditional manual bounding mode, adopts advanced point cloud processing algorithms to automatically identify the target area, and can quickly generate the optimal ROI without manual intervention, significantly improving the consistency of extraction efficiency and accuracy. At the same time, this solution has good adaptability and can meet the requirements of different industrial scenarios, providing reliable technical support for automated detection and intelligent manufacturing, and promoting the transformation and upgrading of industrial production towards the intelligent direction.

[0064] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are some, but not all, of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the protection scope of this application.

[0065] The following disclosure provides many different embodiments or examples for implementing different structures of the present invention. To simplify the disclosure of the present invention, the components and settings of specific examples are described below. Of course, they are only examples and are not intended to limit the present invention. In addition, the present invention may repeat reference numerals and / or letters in different examples. This repetition is for the purpose of simplicity and clarity and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0066] See Figure 1 , which is a schematic flowchart of a method for extracting a region of interest from three-dimensional point cloud data provided by an embodiment of the present application. The method includes:

[0067] S101. Obtain the original three-dimensional point cloud data;

[0068] In the present application, the original three-dimensional point cloud data is data obtained by a structured light three-dimensional camera or the like. The original three-dimensional point cloud data includes the target object and the surrounding environment information. The target object is the object of interest that needs to be labeled by 3D ROI. The target object can be a belt or other objects, and is not specifically limited here. In this embodiment, only the target object being the target belt is taken as an example for illustration. If the target object is the target belt, then in the present application, the original three-dimensional point cloud data of the industrial scene obtained by the structured light three-dimensional camera includes the three-dimensional point cloud on the surface of the target belt, as well as a large number of background objects and outlier noise points in the industrial scene, forming a complex three-dimensional scene.

[0069] S102. Extract the target point cloud data of the target object from the original three-dimensional point cloud data;

[0070] Since the original three-dimensional point cloud data includes not only the point cloud data of the target object, but also the point cloud data of the background objects and outlier noise, this solution needs to extract the target point cloud data of the target object from the original three-dimensional point cloud data. The present application can specifically apply a point cloud segmentation algorithm based on the Euclidean distance to accurately extract the point cloud of the target object by calculating the spatial distance matrix and setting a reasonable threshold.

[0071] S103. Determine the boundary lines of the target object on the target plane according to the target point cloud data, and determine the original endpoint coordinates according to each boundary line;

[0072] In this application, after obtaining the target point cloud data by segmenting the point cloud of the target object, it is necessary to further extract the boundary lines of the target object. In this application, the boundary line is the boundary line of the target object on the target plane, and the number of boundary lines is related to the shape of the target plane. For example, if the target object is a cuboid with six planes and the top surface of the target object is used as the target plane, the boundary lines are the four boundary lines of the top surface of the target object. Further, this application needs to calculate the pairwise intersections of the four boundary lines in sequence and solve the corresponding intersection coordinates, and these intersection coordinates are the original endpoint coordinates of the target object on the target plane. For example, if four boundary lines are recognized, since there are four intersections of the four boundary lines, the coordinates of the four intersections are used as the original endpoint coordinates.

[0073] S104. Use the target rotation angle to convert each original endpoint coordinate into a first endpoint coordinate, and determine the second endpoint coordinate according to the coordinate values of each first endpoint coordinate; wherein, the target rotation angle is the angle between a boundary line and the corresponding coordinate axis direction vector, and the area formed by each second endpoint coordinate is a standard rectangular area;

[0074] It should be noted that the original endpoint coordinates calculated in the above process are the coordinates in the target object coordinate system. To avoid the influence of lens distortion and perspective projection on the original endpoint coordinates, direct processing may lead to errors in the ROI. Therefore, when determining the ROI in this application, the original endpoint coordinates can be converted from the target object coordinate system to the camera coordinate system, and after determining the ROI in the camera coordinate system, it is then converted back to the target object coordinate system.

[0075] When converting in this application, first, it is necessary to determine the target rotation angle, which is the angle between one of the boundary lines and the corresponding coordinate axis direction vector; then, construct a rotation matrix around the Z-axis (vertical axis) according to this target rotation angle, and this rotation matrix is used to transform each original endpoint coordinate into the corresponding first endpoint coordinate. For example, if the four original endpoint coordinates of the target belt are obtained, then apply this rotation matrix for transformation to align the four original endpoint coordinates from the belt coordinate system to the camera coordinate system. After the rotation transformation, the four first endpoint coordinates of the target belt in the corrected coordinate system are obtained.

[0076] Further, after obtaining the first endpoint coordinates of the target object in the camera coordinate system in this application, the second endpoint coordinate can be re-determined according to the coordinate values of the first endpoint coordinates. The second endpoint coordinate can be determined by the maximum value of the horizontal axis (X-axis maximum value), the minimum value of the horizontal axis (X-axis minimum value), the maximum value of the vertical axis (Y-axis maximum value), and the minimum value of the vertical axis (Y-axis minimum value) of each first endpoint coordinate. Through the re-determined second endpoint coordinate, a new boundary range can be determined to form a standard rectangular area, and this standard rectangular area is the maximum region of interest of the target object in the camera coordinate system.

[0077] S105. Convert each second endpoint coordinate into a target endpoint coordinate by using the target rotation angle;

[0078] It should be noted that the second endpoint coordinates determined through the above process are coordinates in the camera coordinate system. In this application, it is necessary to convert each second endpoint coordinate into a target endpoint coordinate by using the target rotation angle, so as to restore from the camera coordinate system to the target object coordinate system.

[0079] S106. Determine a target quadrilateral region according to each target endpoint coordinate, and determine the largest rectangular region within the target quadrilateral region;

[0080] After determining the converted target endpoint coordinates in this application, the target quadrilateral region can be determined according to the target endpoint coordinates. Since the region composed of each second endpoint coordinate determined in the camera coordinate system is a standard rectangular region, the target quadrilateral region determined by the converted target endpoint coordinates is specifically a parallelogram. Then, determine the largest rectangular region within the target quadrilateral region, and this largest rectangular region is the largest rectangular region containing the target object.

[0081] S107. Determine the region of interest by using the largest rectangular region and the height of the target object.

[0082] In this application, the largest rectangular region is the largest rectangular region of the target object on the target plane. For the Z-axis numerical range of the largest rectangular region, it can be determined according to the minimum Z value in the largest rectangular region and the height of the target object. For example, the Z-axis range is: minimum Z value ~ (minimum Z value + height of the target object). It can be seen that the region determined by the horizontal and vertical coordinate ranges of the largest rectangular region and the Z-axis range in this application is the largest 3D ROI.

[0083] In summary, through the automated point cloud processing algorithm, this solution can quickly identify and accurately calculate the largest 3D ROI region, significantly improving the extraction efficiency and accuracy consistency, and avoiding the deficiencies of manual intervention; at the same time, this solution has good adaptability and can meet the requirements of different industrial scenarios, providing reliable support for automated detection and intelligent manufacturing, and promoting the intelligent transformation and upgrading of industrial production. Through this solution, the problems of low efficiency and unstable accuracy caused by relying on manual operation in the prior art are effectively solved.

[0084] In another embodiment of this application, the process of determining the boundary straight line of the target object on the target plane according to the target point cloud data and determining the original endpoint coordinates according to each boundary straight line specifically includes the following steps: Extract the boundary point cloud data of the target object from the target point cloud data; Perform linear fitting on the boundary point cloud data to determine the boundary straight line of the target object on the target plane; Use the coordinate of the intersection point of the straight lines between the boundary straight lines as the original endpoint coordinates.

[0085] It should be noted that when determining the original endpoint coordinates in this application, it is necessary to extract the boundary point cloud data of the target object from the target point cloud data. In another embodiment of this application, by determining the normal vectors of each data point in the target point cloud data, calculating the included angle between the normal vectors of adjacent data points in the target point cloud data, and determining the boundary point cloud data according to the included angle between the normal vectors and the angle threshold.

[0086] Specifically, the target point cloud data in this embodiment can be expressed as: , where the i-th data point has three-dimensional coordinates: . In order to accurately detect the boundary point cloud data in this application, it is necessary to first calculate the normal vector of each point. The estimation of the normal vector is usually based on the principal component analysis method, and is solved by constructing the local neighborhood of the point and calculating the covariance matrix.

[0087] For the point , its set of K-nearest neighbor points is defined as: ;

[0088] where represents the set of K-nearest neighborhoods of the point , is the first K points sorted in ascending order of the distance from the point , that is, satisfying:

[0089]

[0090] where represents the Euclidean distance between two points.

[0091] The covariance matrix is calculated as follows: ;

[0092] where is the local neighborhood center of : ;

[0093] Perform eigenvalue decomposition on the covariance matrix: ;

[0094] where is the eigenvalue, is the corresponding eigenvector. The eigenvector corresponding to the smallest eigenvalue is the normal vector of this point.

[0095] After obtaining the normal vectors of all data points, calculate the included angle between the normal vectors of adjacent data points to judge the boundary information of the point cloud. For the point and its neighbor point , the included angle of the normal vectors is calculated by the following formula: .

[0096] Among them, represents the normal vector of represents the normal vector of represents the dot product of the normal vectors, represents the product of the magnitudes of the normal vectors. If is greater than the set angle threshold , then this point may belong to the boundary point.

[0097] In this application, in order to further improve the robustness of detection, a statistical method is used to determine whether the point is a boundary point. Define the boundary feature score of the point as:

[0098]

[0099] Among them, is an indicator function. If then it returns 1, otherwise it returns 0. If exceeds the set threshold, then it is considered that the point is a boundary point; that is: select each neighbor point from the set of K-nearest neighbor points of the data point . If the total number of the included angles between the normal vectors of the data point and the neighbor points that exceed the angle threshold exceeds the set threshold, then it is determined that the data point is a boundary point.

[0100] Through the above process, this application can determine the boundary point cloud data from the target point cloud data. In another embodiment of this application, the process of performing a straight line fitting on the boundary point cloud data to determine the boundary straight line of the target object on the target plane includes: determining each target straight line from the boundary point cloud data; the target straight line is determined by at least two target data points in the boundary point cloud data; determining the boundary straight line that forms an acute angle with the target axis direction vector from each target straight line; among them, the target axis direction vector includes the horizontal axis direction vector and the vertical axis direction vector; the boundary straight line includes the upper boundary straight line, the lower boundary straight line, the left boundary straight line, and the right boundary straight line of the target object on the target plane.

[0101] Specifically, after this application extracts the boundary point cloud data of the target object, it is necessary to perform a straight line fitting on the boundary points and sequentially extract the upper, lower, left, and right four boundary straight lines of the target object. Therefore, when performing a straight line fitting in this application, it is necessary to sequentially fit two straight lines that form an acute angle with the horizontal axis direction vector, and two straight lines that form an acute angle with the vertical axis direction vector. Here, the specific steps of performing a straight line fitting on this application are described:

[0102] a. Randomly sample the boundary point cloud data, set the threshold to 1, and obtain the best straight line. The following are the steps to fit the first straight line with an acute angle between the direction vector (1, 0, 0):

[0103] Randomly select the minimum number of points from the data; among them, when fitting a straight line in this application, at least 2 target data points are selected.

[0104] Calculate the model through at least two target data points, and this model is the target straight line; and calculate whether the angle between the target straight line and the direction vector (1, 0, 0) (the horizontal axis direction vector) is an acute angle, and retain the target straight line in the case of an acute angle.

[0105] Then calculate the distances from all data points to this model (the target straight line), and count the points that meet the distance threshold; in this application, specifically, count the data points with a distance less than 1 from the target straight line, and call this data point an inlier.

[0106] In this application, for each target straight line, it is necessary to calculate the inliers of each target straight line through the above process. If the number of inliers of the current target straight line is more than that of the previous best model, then use the current target straight line as the best model, repeat the above steps several times, and finally select the best model with the largest number of inliers as the best fit, which is the boundary straight line in this application.

[0107] b. After obtaining the first boundary straight line through step a, it is necessary to calculate the distance of each point to this straight line : ;

[0108] where is a point on the straight line, and d is the unit direction vector of the straight line. If is less than the set threshold, then it is regarded as an inlier and this point needs to be removed.

[0109] c. After removing the point cloud on and near the fitted boundary straight line, the remaining point cloud is again fitted in the manner of steps a - b. After fitting, generate another boundary straight line with an acute angle between the direction vector (1, 0, 0), and remove the point cloud on and near the fitted boundary straight line to obtain the screened point cloud data.

[0110] d. Apply the random sampling algorithm to the screened point cloud data, but this time the screening target is a straight line with an acute angle between the direction vector (0, 1, 0) (the vertical axis direction vector), and obtain two boundary straight lines with an acute angle between the direction vector (0, 1, 0).

[0111] e. After the above steps, the equations of four boundary straight lines can be finally obtained.

[0112] As can be seen from the above, when determining the original endpoint coordinates in this application, the boundary point cloud data can be specifically determined according to the normal vector angles and angle thresholds of the data points in the target point cloud data; in this boundary point cloud data, two straight lines that are acute angles with the horizontal axis direction vector and two straight lines that are acute angles with the vertical axis direction vector are fitted by a straight line fitting method, and the above four fitted straight lines are used as the boundary straight lines of the target object. Through this method, the boundary straight lines of the target object can be quickly and accurately found in this application, and then the original endpoint coordinates can be determined through the straight line intersection coordinates of the boundary straight lines.

[0113] In another embodiment of this application, the process of converting each original endpoint coordinate into a first endpoint coordinate by using the target rotation angle includes:

[0114] Determine the target boundary straight line from the boundary straight lines; the angle between the target boundary straight line and the horizontal axis direction vector is an acute angle; use the angle between the target boundary straight line and the horizontal axis direction vector as the target rotation angle; determine the rotation matrix by using the target rotation angle, and convert each original endpoint coordinate into a first endpoint coordinate through the rotation matrix.

[0115] In this application, four boundary straight lines are obtained through fitting by the above process. When determining the target rotation angle in this application, specifically, the target boundary straight line with an acute angle with the horizontal axis direction vector (X-axis direction vector (1, 0, 0)) can be found from the four boundary straight lines; in this embodiment, the first boundary straight line determined by random sampling in the above embodiment can be used as the target boundary straight line. Then calculate the angle between the target boundary straight line and the horizontal axis direction vector. This application calls this angle the target rotation angle, and this target rotation angle reflects the offset angle of the target object coordinate system relative to the camera coordinate system; after determining the rotation matrix through the target rotation angle in this application, each original endpoint coordinate can be converted into a first endpoint coordinate through the rotation matrix.

[0116] In another embodiment of this application, after converting each original endpoint coordinate into a first endpoint coordinate by the rotation matrix in this application, it is also necessary to determine the maximum value of the horizontal axis, the minimum value of the horizontal axis, the maximum value of the vertical axis, and the minimum value of the vertical axis according to each first endpoint coordinate; determine the second endpoint coordinates according to the maximum value of the horizontal axis, the minimum value of the horizontal axis, the maximum value of the vertical axis, and the minimum value of the vertical axis; among them, the vertical axis coordinate value of each second endpoint coordinate is the minimum vertical axis coordinate value of each first endpoint coordinate.

[0117] That is to say, after the present application converts each original endpoint coordinate into the camera coordinate system through the rotation matrix, the maximum rectangle can be determined in the camera coordinate system; in this embodiment, it is necessary to determine the minimum X value (the minimum value on the horizontal axis), the maximum X value (the maximum value on the horizontal axis), the minimum Y value (the minimum value on the vertical axis), and the maximum Y value (the maximum value on the vertical axis) according to each first endpoint coordinate, and construct four new endpoints through the above values to enclose a standard rectangle area, and uniformly set the Z coordinate value (the vertical axis coordinate value) of all endpoints to the minimum Z value (the minimum vertical axis coordinate value) among the four endpoints to ensure that the standard rectangle area is in the same plane. At this time, this standard rectangle area is the maximum 3D ROI after the target object coordinate system is converted to the camera coordinate system.

[0118] After the present application determines each second endpoint coordinate, it is also necessary to reverse-rotate each second endpoint coordinate through the rotation matrix determined by the target rotation angle to restore it from the camera coordinate system to the target object coordinate system, so as to obtain the maximum parallelogram area of the target object.

[0119] In summary, in the present application, in order to obtain the most accurate maximum rectangle area of the target object, the present application can convert the original endpoint coordinates from the target object coordinate system to the camera coordinate system according to the target rotation angle, and determine the maximum rectangle area of the target object in the camera coordinate system through the coordinate values of each endpoint. And the maximum rectangle area can be in the same plane by unifying the vertical axis coordinate values. Then, the endpoint coordinates of the standard rectangle area are reverse-rotated through the target rotation angle to restore it from the camera coordinate system to the target object coordinate system, so as to obtain the largest parallelogram area on the target object.

[0120] In another embodiment of the present application, the process of determining the maximum rectangle area within the target quadrilateral area specifically includes the following contents: within the horizontal and vertical axis coordinate ranges of the target quadrilateral area, determine each current center point; determine candidate vertices according to each current center point; determine whether each candidate vertex is located inside the target quadrilateral area; if so, determine a candidate rectangle area according to each candidate vertex; and use the candidate rectangle area with the largest area as the maximum rectangle area.

[0121] In this embodiment, after determining the target quadrilateral area of the target object through the above embodiment, the X-axis coordinate value range and the Y-axis coordinate value range of the target quadrilateral area can be calculated, and the maximum and minimum values of the X-axis coordinate value range and the Y-axis coordinate value range are rounded to ensure that the area boundary is aligned to integer coordinates, which is convenient for subsequent processing and calculation.

[0122] The present application needs to find the maximum rectangle area that can be completely contained within the target quadrilateral area. The specific search process includes the following steps:

[0123] a. Traversal search: Traverse within the X-axis coordinate value range and Y-axis coordinate value range of the parallelogram to determine the current point, and check possible rectangular areas point by point with an integer step size.

[0124] b. Construct four candidate vertices of the rectangle: For the current point , calculate the corresponding four candidate vertices based on central symmetry to form a candidate rectangle.

[0125] c. Calculate the cross product: For each side of the quadrilateral (e.g., side ), calculate the cross product from any candidate vertex to each side. In this application, any candidate vertex is represented by point , and the cross product formula is as follows:

[0126] ;

[0127] where, is the x-coordinate of point A, is the x-coordinate of point B, is the x-coordinate of point P, is the y-coordinate of point A, is the y-coordinate of point B, is the y-coordinate of point P.

[0128] d. Check the cross product sign: Calculate the cross products of each candidate vertex with the four sides of the target quadrilateral and check the sign of each cross product. If the four cross products of a certain candidate vertex are all greater than or equal to 0, or all less than or equal to 0, it means that the candidate vertex is inside the target quadrilateral. Otherwise, the candidate vertex is outside the target quadrilateral; in this application, select the four candidate vertices that are all inside the target quadrilateral to form a candidate rectangular area.

[0129] e. Calculate the rectangle area: For the candidate rectangular areas that meet the conditions, calculate their areas and compare them with the current maximum area to update the position information of the maximum candidate rectangular area.

[0130] f. Final result: After the traversal is completed, the candidate rectangular area with the largest area found is the largest rectangular area inside the target quadrilateral.

[0131] Through the above process, this application can determine the largest rectangular area of the target object on the X-axis and Y-axis, and then determine the Z-axis range of the largest rectangular area by the minimum Z value in the largest rectangular area and the height of the target object. The finally calculated area is the largest 3D ROI.

[0132] In summary, the present application proposes an intelligent maximum 3D ROI extraction solution based on three-dimensional vision. This solution analyzes and processes the point cloud of the target object, and accurately calculates the maximum enclosed 3D ROI area on the premise that the XY coordinate axes of the ROI are parallel to the XY axes of the camera coordinate system. This method calculates completely relying on the belt point cloud data without manual intervention, can efficiently extract the maximum three-dimensional ROI of the workpiece area, improve the extraction accuracy and efficiency, and make it more suitable for automated detection and intelligent manufacturing.

[0133] See Figure 2 , Figure 2 which is a schematic structural diagram of a device for extracting an interested region of three-dimensional point cloud data provided by an embodiment of the present application. The device specifically includes:

[0134] An acquisition module 11, configured to acquire original three-dimensional point cloud data;

[0135] An extraction module 12, configured to extract target point cloud data of a target object from the original three-dimensional point cloud data;

[0136] A first determination module 13, configured to determine boundary lines of the target object on a target plane according to the target point cloud data, and determine original endpoint coordinates according to the boundary lines;

[0137] A first conversion module 14, configured to convert each original endpoint coordinate into a first endpoint coordinate by using a target rotation angle, and determine a second endpoint coordinate according to the coordinate values of each first endpoint coordinate; wherein, the target rotation angle is the included angle between the boundary line and the corresponding coordinate axis direction vector, and the area formed by each second endpoint coordinate is a standard rectangular area;

[0138] A second conversion module 15, configured to convert each second endpoint coordinate into a target endpoint coordinate by using the target rotation angle;

[0139] A second determination module 16, configured to determine a target quadrilateral area according to each target endpoint coordinate, and determine a maximum rectangular area within the target quadrilateral area;

[0140] A third determination module 17, configured to determine an interested region by using the maximum rectangular area and the height of the target object.

[0141] As an optional embodiment, the first determination module includes:

[0142] An extraction unit, configured to extract boundary point cloud data of the target object from the target point cloud data;

[0143] A fitting unit, configured to perform linear fitting on the boundary point cloud data to determine the boundary lines of the target object on the target plane;

[0144] A coordinate determination unit, configured to use the coordinates of the intersection points of the straight lines between the boundary straight lines as the original endpoint coordinates.

[0145] As an optional embodiment, the extraction unit is specifically configured to: determine the normal vectors of the respective data points in the target point cloud data, calculate the included angles between the normal vectors of adjacent data points in the target point cloud data, and determine the boundary point cloud data according to the included angle between the normal vectors and the angle threshold.

[0146] As an optional embodiment, the fitting unit is specifically configured to: determine respective target straight lines from the boundary point cloud data; the target straight lines are determined by at least two target data points in the boundary point cloud data; determine boundary straight lines that form acute angles with the target axis direction vectors from the respective target straight lines; wherein the target axis direction vectors include a horizontal axis direction vector and a vertical axis direction vector; the boundary straight lines include the upper boundary straight line, the lower boundary straight line, the left boundary straight line, and the right boundary straight line of the target object in the target plane.

[0147] As an optional embodiment, the first conversion module includes:

[0148] A straight line determination unit, configured to determine a target boundary straight line from the boundary straight lines; the included angle between the target boundary straight line and the horizontal axis direction vector is an acute angle;

[0149] An included angle determination unit, configured to use the included angle between the target boundary straight line and the horizontal axis direction vector as the target rotation angle;

[0150] A matrix determination unit, configured to determine a rotation matrix using the target rotation angle;

[0151] A conversion unit, configured to convert each original endpoint coordinate into a first endpoint coordinate through the rotation matrix.

[0152] As an optional embodiment, the first conversion module further includes:

[0153] A coordinate value determination unit, configured to determine a horizontal axis maximum value, a horizontal axis minimum value, a vertical axis maximum value, and a vertical axis minimum value according to each first endpoint coordinate;

[0154] An endpoint determination unit, configured to determine second endpoint coordinates according to the horizontal axis maximum value, the horizontal axis minimum value, the vertical axis maximum value, and the vertical axis minimum value; wherein the vertical axis coordinate value of each second endpoint coordinate is the minimum vertical axis coordinate value of each first endpoint coordinate.

[0155] As an optional embodiment, the second determination module includes:

[0156] A center point determination unit, configured to determine respective current center points within the horizontal and vertical axis coordinate ranges of the target quadrilateral region;

[0157] A candidate vertex determination unit, configured to determine candidate vertices according to each current center point;

[0158] A judgment unit, configured to judge whether each candidate vertex is located inside the target quadrilateral region; if so, determine a candidate rectangular region according to each candidate vertex;

[0159] A region determination unit, configured to use the candidate rectangular region with the largest area as the largest rectangular region.

[0160] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0161] See Figure 3 , Figure 3 FIG. is a schematic structural diagram of an electronic device provided by an embodiment of the present application, including a processor 21, a communication interface 22, a memory 23, and a communication bus 24. Among them, the processor 21, the communication interface 22, and the memory 23 complete communication with each other through the communication bus 24;

[0162] The memory 23 is used to store a computer program;

[0163] When the processor 21 is used to execute the program stored in the memory 23, it implements the steps of the method for extracting an area of interest described in any of the above method embodiments, which will not be elaborated herein.

[0164] The communication bus mentioned in the above terminal may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity of representation, Figure 3 only a thick line is shown in, but it does not mean that there is only one bus or one type of bus.

[0165] The communication interface is used for communication between the above terminal and other devices.

[0166] The memory may include a Random Access Memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0167] The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP for short), an application specific integrated circuit (ASIC for short), a field-programmable gate array (FPGA for short), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0168] In another exemplary embodiment, a computer storage medium is also provided. When the program instructions are executed by a processor, the steps of the region of interest extraction method described in any of the above method embodiments are implemented. Among them, the storage medium may include: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc.

[0169] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and will not be elaborated here.

[0170] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" as used herein may also include the plural forms. The terms "include", "comprise", "contain", and "have" are inclusive and thus specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring them to be executed in the particular order described or illustrated, unless the execution order is explicitly stated. It should also be understood that additional or alternative steps may be used.

[0171] The above are only specific embodiments of the present invention, enabling those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for extracting a region of interest from three-dimensional point cloud data, characterized in that, Including: Obtain the original three-dimensional point cloud data; Extract the target point cloud data of the target object from the original three-dimensional point cloud data; According to the target point cloud data, determine the boundary lines of the target object on the target plane, and determine the original endpoint coordinates according to each boundary line; Use the target rotation angle to convert each original endpoint coordinate into a first endpoint coordinate, and determine the second endpoint coordinate according to the coordinate values of each first endpoint coordinate; wherein, the target rotation angle is the included angle between the boundary line and the corresponding coordinate axis direction vector, and the area formed by each second endpoint coordinate is a standard rectangular area; Use the target rotation angle to convert each second endpoint coordinate into a target endpoint coordinate; Determine the target quadrilateral area according to each target endpoint coordinate, and determine the largest rectangular area within the target quadrilateral area; Use the largest rectangular area and the height of the target object to determine the region of interest; Among them, the step of using the target rotation angle to convert each original endpoint coordinate into a first endpoint coordinate includes: Determine the target boundary line from the boundary lines; the included angle between the target boundary line and the horizontal axis direction vector is an acute angle; Take the included angle between the target boundary line and the horizontal axis direction vector as the target rotation angle; Determine the rotation matrix using the target rotation angle; Convert each original endpoint coordinate into a first endpoint coordinate through the rotation matrix; Among them, the step of determining the second endpoint coordinate according to the coordinate values of each first endpoint coordinate includes: Determine the maximum value of the horizontal axis, the minimum value of the horizontal axis, the maximum value of the vertical axis, and the minimum value of the vertical axis according to each first endpoint coordinate; Determine the second endpoint coordinate according to the maximum value of the horizontal axis, the minimum value of the horizontal axis, the maximum value of the vertical axis, and the minimum value of the vertical axis; wherein, the vertical axis coordinate value of each second endpoint coordinate is the minimum vertical axis coordinate value of each first endpoint coordinate.

2. The method for extracting the region of interest according to claim 1, wherein According to the target point cloud data, determining the boundary lines of the target object on the target plane, and determining the original endpoint coordinates according to each boundary line includes: Extract the boundary point cloud data of the target object from the target point cloud data; Perform line fitting on the boundary point cloud data to determine the boundary lines of the target object on the target plane; Take the straight line intersection point coordinates between the boundary lines as the original endpoint coordinates.

3. The method for extracting a region of interest according to claim 2, wherein Extracting the boundary point cloud data of the target object from the target point cloud data includes: Determine the normal vector of each data point in the target point cloud data; Calculate the included angle between the normal vectors of adjacent data points in the target point cloud data; Determine the boundary point cloud data according to the included angle between the normal vectors and the angle threshold.

4. The method for extracting the region of interest according to claim 2, wherein Performing line fitting on the boundary point cloud data to determine the boundary lines of the target object on the target plane includes: Determine each target line from the boundary point cloud data; the target line is determined by at least two target data points in the boundary point cloud data; Determine the boundary line with an acute angle with the target axis direction vector from each target line; Among them, the target axis direction vector includes the horizontal axis direction vector and the vertical axis direction vector; the boundary lines include the upper boundary line, the lower boundary line, the left boundary line, and the right boundary line of the target object on the target plane.

5. The method for extracting a region of interest according to any one of claims 1 to 4, characterized in that, Determining the largest rectangular area within the target quadrilateral area includes: Within the horizontal and vertical axis coordinate ranges of the target quadrilateral region, determine each current center point; Determine candidate vertices based on each current center point; Judge whether each candidate vertex is located inside the target quadrilateral region; If so, determine a candidate rectangular region based on each candidate vertex; Take the candidate rectangular region with the largest area as the largest rectangular region.

6. An apparatus for extracting a region of interest from three-dimensional point cloud data, characterized in that, Include: An acquisition module for acquiring original three-dimensional point cloud data; An extraction module for extracting target point cloud data of a target object from the original three-dimensional point cloud data; A first determination module for determining boundary lines of the target object on a target plane according to the target point cloud data, and determining original endpoint coordinates according to each boundary line; A first conversion module for converting each original endpoint coordinate into a first endpoint coordinate by using a target rotation angle, and determining a second endpoint coordinate according to the coordinate values of each first endpoint coordinate; wherein, the target rotation angle is the included angle between the boundary line and the corresponding coordinate axis direction vector, and the region formed by each second endpoint coordinate is a standard rectangular region; A second conversion module for converting each second endpoint coordinate into a target endpoint coordinate by using the target rotation angle; A second determination module for determining a target quadrilateral region according to each target endpoint coordinate and determining the largest rectangular region within the target quadrilateral region; A third determination module for determining a region of interest by using the largest rectangular region and the height of the target object; Wherein, the first conversion module includes: A line determination unit for determining a target boundary line from the boundary lines; the included angle between the target boundary line and the horizontal axis direction vector is an acute angle; An included angle determination unit for taking the included angle between the target boundary line and the horizontal axis direction vector as the target rotation angle; A matrix determination unit for determining a rotation matrix by using the target rotation angle; A conversion unit for converting each original endpoint coordinate into a first endpoint coordinate through the rotation matrix; Wherein, the first conversion module further includes: A coordinate value determination unit for determining the maximum value of the horizontal axis, the minimum value of the horizontal axis, the maximum value of the vertical axis, and the minimum value of the vertical axis according to each first endpoint coordinate; An endpoint determination unit for determining a second endpoint coordinate according to the maximum value of the horizontal axis, the minimum value of the horizontal axis, the maximum value of the vertical axis, and the minimum value of the vertical axis; wherein, the vertical axis coordinate value of each second endpoint coordinate is the minimum vertical axis coordinate value of each first endpoint coordinate.

7. An electronic device, characterized in that, Include a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete mutual communication through the communication bus; The memory is used for storing a computer program; When the processor is used to execute the program stored on the memory, it realizes the steps of the method for extracting a region of interest described in any one of claims 1 to 5.

8. A computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, and the computer-executable instructions are used to execute the steps of the method for extracting a region of interest described in any one of claims 1 to 5.

Images