Point cloud processing method, device and equipment for non-unilateral workpiece and storage medium

Abstract

The method, device, equipment and storage medium for processing point clouds of non-unilateral workpieces provided by the present disclosure determine visible point clouds of each point cloud template under the camera field of view according to the attribute information of the template points in the point cloud template and the corresponding viewpoint information of the camera, select the point cloud template corresponding to the workpiece to be processed from all point cloud templates of the workpieces according to the scene point cloud of the workpiece to be processed and the visible point clouds corresponding to the point cloud templates, and then perform fine matching processing on the scene point cloud according to the visible point cloud corresponding to the selected point cloud template. In this technical solution, the camera viewpoint information and the attribute information of each template point are used to screen the visible point clouds under the camera field of view, so that the matching of the scene point cloud and the point cloud template is quickly achieved to increase the efficiency of fine matching.

Description

Technical Field

[0001] This disclosure relates to the field of machine vision technology, and in particular to a method, apparatus, device and storage medium for point cloud processing of non-unilateral workpieces. Background Technology

[0002] In workpiece gripping applications, fine matching obtains the workpiece's pose with low precision from coarse matching, and then gradually improves the workpiece matching accuracy through iteration.

[0003] For scene point clouds captured by the camera, only the upper surface of the workpiece can be captured, while the template point cloud of the model is a complete template point cloud. The inability to quickly find the corresponding template point cloud within the scene negatively impacts both the speed and accuracy of fine matching, thus preventing the rapid achievement of the goal of fine matching. Summary of the Invention

[0004] This disclosure provides a point cloud processing method, apparatus, device, and storage medium for non-unilateral workpieces, to solve the problem that the prior art cannot quickly achieve precise matching between point cloud templates and scene point clouds.

[0005] In a first aspect, embodiments of this disclosure provide a point cloud processing method for a non-unilateral workpiece, including:

[0006] For each workpiece, a point cloud template is provided. Based on the attribute information of the template points in the point cloud template and the corresponding viewpoint information of the camera, the visible point cloud of each point cloud template under the camera's field of view is determined. The attribute information is either vector information or depth information obtained by orthographic projection of the template points.

[0007] Based on the scene point cloud of the workpiece to be processed and the visible point cloud corresponding to each point cloud template, select the point cloud template corresponding to the workpiece to be processed from the point cloud templates of all workpieces.

[0008] Based on the visible point cloud corresponding to the selected point cloud template, the scene point cloud is subjected to fine matching processing.

[0009] In one or more embodiments, the viewpoint information is a viewpoint vector pointing from the camera to the template point, and the attribute information is the normal vector of the template point;

[0010] Accordingly, determining the visible point cloud of each point cloud template under the camera's field of view based on the attribute information of the template points in the point cloud template and the corresponding viewpoint information of the camera includes:

[0011] For each template point, determine the angle between the normal vector of the template point and its corresponding viewpoint vector;

[0012] The set of template points where the angle between all the normal vectors and the corresponding viewpoint vectors is obtuse is taken as the visible point cloud of the point cloud template in the camera's field of view.

[0013] In one or more embodiments, the viewpoint information is the optical axis direction of the camera, and the attribute information is the vertical component of the normal vector of the template point;

[0014] Accordingly, determining the visible point cloud of each point cloud template under the camera's field of view based on the attribute information of the template points in the point cloud template and the corresponding viewpoint information of the camera includes:

[0015] Compare the orientation of the vertical axis component of each template point with the direction of the optical axis;

[0016] The set of template points whose orientation is opposite to that of the optical axis is taken as the visible point cloud of the point cloud template in the camera's field of view.

[0017] In one or more embodiments, the viewpoint information is the optical axis direction of the camera, and the attribute information is the depth information;

[0018] Accordingly, determining the visible point cloud of each point cloud template under the camera's field of view based on the attribute information of the template points in the point cloud template and the corresponding viewpoint information of the camera includes:

[0019] Based on the optical axis direction, orthographic projection is performed on each template point;

[0020] The set of all template points that do not overlap in orthographic projection is taken as the first part of the visible point cloud of the point cloud template in the camera's field of view;

[0021] For at least two template points where orthographic projection overlap occurs, obtain the depth information between each template point and the corresponding orthographically projected point.

[0022] The set of all template points with the largest depth information corresponding to the same orthographic projection is taken as the second part of the visible point cloud of the point cloud template in the camera's field of view.

[0023] In one or more embodiments, determining the visible point cloud of each point cloud template under the camera's field of view based on the attribute information of template points in the point cloud template and the corresponding viewpoint information of the camera includes:

[0024] Determine whether the angle between the normal vector of the template point and the viewpoint vector corresponding to the template point is an acute angle;

[0025] If there exists a template point where the angle between the normal vector and the corresponding viewpoint vector is acute, then the visible point cloud of the point cloud template in the camera's field of view is determined based on the attribute information of the template point in the point cloud template and the corresponding viewpoint information of the camera.

[0026] In one or more embodiments, the method further includes:

[0027] Based on the selected point cloud template and the scene point cloud, other point clouds in the workpiece to be processed are determined.

[0028] Secondly, embodiments of this disclosure provide a point cloud processing apparatus for non-unilateral workpieces, comprising:

[0029] The first determining module is used to determine the visible point cloud of each point cloud template in the field of view of the camera based on the attribute information of the template points in the point cloud template and the corresponding viewpoint information of the camera, for each point cloud template corresponding to each workpiece. The attribute information is vector information or depth information obtained by orthographic projection of the template points.

[0030] The second determining module is used to select the point cloud template corresponding to the workpiece to be processed from the point cloud templates of all workpieces, based on the scene point cloud of the workpiece to be processed and the visible point cloud corresponding to each point cloud template.

[0031] The processing module is used to perform fine matching processing on the scene point cloud based on the visible point cloud corresponding to the selected point cloud template.

[0032] In one or more embodiments, the viewpoint information is a viewpoint vector pointing from the camera to the template point, and the attribute information is the normal vector of the template point;

[0033] Accordingly, the first determining module is specifically used for:

[0034] For each template point, determine the angle between the normal vector of the template point and its corresponding viewpoint vector;

[0035] The set of template points where the angle between all the normal vectors and the corresponding viewpoint vectors is obtuse is taken as the visible point cloud of the point cloud template in the camera's field of view.

[0036] In one or more embodiments, the viewpoint information is the optical axis direction of the camera, and the attribute information is the vertical component of the normal vector of the template point;

[0037] Accordingly, the first determining module is specifically used for:

[0038] Compare the orientation of the vertical axis component of each template point with the direction of the optical axis;

[0039] The set of template points whose orientation is opposite to that of the optical axis is taken as the visible point cloud of the point cloud template in the camera's field of view.

[0040] In one or more embodiments, the viewpoint information is the optical axis direction of the camera, and the attribute information is the depth information;

[0041] Accordingly, the first determining module is specifically used for:

[0042] Based on the optical axis direction, orthographic projection is performed on each template point;

[0043] The set of all template points that do not overlap in orthographic projection is taken as the first part of the visible point cloud of the point cloud template in the camera's field of view;

[0044] For at least two template points where orthographic projection overlap occurs, obtain the depth information between each template point and the corresponding orthographically projected point.

[0045] The set of all template points with the largest depth information corresponding to the same orthographic projection is taken as the second part of the visible point cloud of the point cloud template in the camera's field of view.

[0046] In one or more embodiments, the viewpoint information is a viewpoint vector, the viewpoint vector is a vector pointing from the camera to the template point, and the attribute information is vector information, the vector information is the normal vector of the template point;

[0047] Accordingly, before determining the visible point cloud of the point cloud template under the camera's field of view based on the attribute information of each template point in the point cloud template and the corresponding viewpoint information of the camera, the first determining module is further configured to:

[0048] Determine whether the angle between the normal vector of the template point and the viewpoint vector corresponding to the template point is an acute angle;

[0049] If there exists a template point where the angle between the normal vector and the corresponding viewpoint vector is acute, then the visible point cloud of the point cloud template in the camera's field of view is determined based on the attribute information of the template point in the point cloud template and the corresponding viewpoint information of the camera.

[0050] In one or more embodiments, the first determining module is further configured to:

[0051] Based on the selected point cloud template and the scene point cloud, other point clouds in the workpiece to be processed are determined.

[0052] Thirdly, this disclosure provides an electronic device, including: a processor, and a memory and a transceiver communicatively connected to the processor;

[0053] The memory stores computer-executed instructions; the transceiver is used for sending and receiving data.

[0054] The processor executes computer execution instructions stored in the memory to implement the method as described in the first aspect or any of the above methods.

[0055] Fourthly, this disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in the first aspect or any of the above methods.

[0056] Fifthly, this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect or any of the above methods.

[0057] This disclosure provides a point cloud processing method, apparatus, device, and storage medium for non-unilateral workpieces. The method involves determining the visible point cloud of each point cloud template within the camera's field of view based on the attribute information of the template points and the corresponding viewpoint information of the camera, for each workpiece. Then, based on the scene point cloud of the workpiece to be processed and the visible point clouds corresponding to each template, a point cloud template corresponding to the workpiece is selected from all the workpiece's point cloud templates. Finally, the scene point cloud is finely matched based on the visible point cloud corresponding to the selected template. This technical solution utilizes camera viewpoint information and the attribute information of each template point to filter the visible point cloud within the camera's field of view, thereby quickly achieving matching between the scene point cloud and the point cloud template, increasing the efficiency of fine matching. Attached Figure Description

[0058] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0059] Figure 1 A flowchart illustrating the point cloud processing method for non-unilateral workpieces provided in this embodiment of the disclosure. Figure 1 ;

[0060] Figure 2 A flowchart illustrating the point cloud processing method for non-unilateral workpieces provided in this embodiment of the disclosure. Figure 2 ;

[0061] Figure 3 Schematic diagram of the determination of visible point clouds Figure 1 ;

[0062] Figure 4 A flowchart illustrating the point cloud processing method for non-unilateral workpieces provided in this embodiment of the disclosure. Figure 3 ;

[0063] Figure 5 Schematic diagram of the determination of visible point clouds Figure 2 ;

[0064] Figure 6 A flowchart illustrating the point cloud processing method for non-unilateral workpieces provided in this embodiment of the disclosure. Figure 4 ;

[0065] Figure 7 Schematic diagram of the determination of visible point clouds Figure 3 ;

[0066] Figure 8 A schematic diagram of the point cloud processing device for a non-unilateral workpiece provided in an embodiment of this disclosure;

[0067] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0068] The accompanying drawings have illustrated specific embodiments of this disclosure, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this disclosure to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0070] Before introducing the embodiments of this disclosure, the application background of the embodiments of this disclosure will first be explained:

[0071] In workpiece gripping applications, fine matching obtains the workpiece's pose with low precision from coarse matching, and then gradually improves the workpiece matching accuracy through iteration.

[0072] For scene point clouds captured by the camera, only the upper surface of the workpiece can be captured, while the template point cloud of the model is a complete template point cloud. The inability to quickly find the corresponding template point cloud within the scene negatively impacts both the speed and accuracy of fine matching, thus preventing the rapid achievement of the goal of fine matching.

[0073] Therefore, the technical problem to be solved by the embodiments of this disclosure is: how to quickly determine the point cloud template required for fine matching of the scene point cloud of the workpiece to be measured.

[0074] To address the technical problems existing in the prior art, the inventors of this disclosure have the following concept: When actually observing the point cloud in the point cloud template, it was found that the angle between the normal vector of each template point in the visible area of ​​the point cloud template under the camera's field of view and its respective viewpoint vector is an obtuse angle; the Z-axis vector of the normal vector points in the opposite direction to the optical axis of the camera; and when orthographically projecting each template point, the template point through which a beam of orthographically projected light first passes can be a template point in the visible area of ​​the point cloud template. Therefore, it is possible to delete the invisible point cloud in the point cloud template under the camera's field of view based on the attribute information of the template points and the viewpoint information of the camera to obtain the visible point cloud. This allows for rapid matching of the scene point cloud of the workpiece to be measured to obtain the corresponding point cloud template, thereby improving the efficiency of fine matching.

[0075] The technical solutions of this disclosure will now be described in detail through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0076] It is worth noting that the application fields of the point cloud processing method, apparatus, equipment and storage medium for non-unilateral workpieces disclosed herein are not limited.

[0077] In this disclosure, the executing entity is an electronic device, specifically a control unit in the electronic device or a controller that controls the electronic device.

[0078] Figure 1 A flowchart illustrating the point cloud processing method for non-unilateral workpieces provided in this embodiment of the disclosure. Figure 1 ,like Figure 1 As shown, the point cloud processing method for a non-unilateral workpiece may include the following steps:

[0079] Step 11: For each workpiece, based on the attribute information of the template points in the point cloud template and the corresponding viewpoint information of the camera, determine the visible point cloud of each point cloud template in the camera's field of view.

[0080] The attribute information is either vector information or depth information obtained by orthographic projection onto the template point.

[0081] In this solution, a non-unilateral workpiece can be a workpiece with at least a partial structure at the bottom in any placement scenario, that is, the bottom structure of the workpiece cannot be captured from the camera's field of view.

[0082] In this step, since the scene point cloud obtained by the camera represents the point cloud of the visible area of ​​the workpiece to be measured under the camera's field of view, in order to perform relevant processing for fine matching of the scene point cloud, it is necessary to determine the point cloud template required for fine matching of the workpiece to be measured from the point cloud templates corresponding to different workpieces in advance.

[0083] Therefore, it is necessary to preprocess the point cloud template first, that is, to remove the invisible point cloud in the point cloud template that is not visible in the camera's field of view, that is, to remove the invisible points in the point cloud model.

[0084] Optionally, for each point cloud template, invisible points in the point cloud template can be removed based on the camera's corresponding viewpoint information and the attribute information of each template point in the point cloud template, so as to obtain the visible point cloud of the point cloud template under the camera's field of view.

[0085] In this implementation, the attribute information of the template point can be the vector information of the template point, such as the normal vector; or it can be the depth information obtained by orthographic projection of the template point, such as the depth information between the template point and the corresponding orthographically projected point.

[0086] Optionally, one implementation method is to determine whether the angle between the normal vector of the template point and the view vector corresponding to the template point is an acute angle. If there is a template point whose normal vector and the corresponding view vector are acute angles, then the visible point cloud of the point cloud template in the camera's field of view is determined based on the attribute information of the template point in the point cloud template and the corresponding view information of the camera.

[0087] It should be understood that the principle behind this implementation can be referenced below. Figure 3 This will not be elaborated upon here.

[0088] Optionally, the method for determining the point cloud template corresponding to the workpiece can be: measuring the workpiece using a three-dimensional measuring device to obtain the point cloud template corresponding to the workpiece.

[0089] In this implementation, a 3D measuring device, such as a scanner, can perform an all-around scan of the workpiece to obtain the complete point cloud distribution of the workpiece, that is, the point cloud template corresponding to the workpiece.

[0090] Step 12: Based on the scene point cloud of the workpiece to be processed and the visible point cloud corresponding to each point cloud template, select the point cloud template corresponding to the workpiece to be processed from all the point cloud templates of the workpiece.

[0091] In this step, after obtaining the visible point cloud corresponding to each point cloud template, the visible point cloud corresponding to each point cloud template can be roughly compared one by one according to the scene point cloud to select the point cloud template corresponding to the visible point cloud that matches the workpiece to be processed.

[0092] In one possible implementation, a matching threshold can be preset. When the matching degree between the scene point cloud and a certain point cloud template is greater than the matching degree threshold, the point cloud template is selected as the point cloud template. For example, the matching degree threshold can be set to 90%.

[0093] Step 13: Perform fine matching processing on the scene point cloud based on the visible point cloud corresponding to the selected point cloud template.

[0094] In this step, after the point cloud template for fine matching was determined in the above steps, the scene point cloud is finely matched based on the visible point cloud corresponding to the point cloud template.

[0095] Optionally, this process can use point cloud registration techniques, and the general process can be as follows:

[0096] Using appropriate point cloud processing libraries or tools, such as Open3D, perform preliminary, approximate alignment of the visible point cloud and the scene point cloud. Simple registration algorithms, such as the coarse Iterative Closest Point (ICP) algorithm, can be used to align the template point cloud with the scene point cloud as closely as possible. Feature points are then extracted from both sets of point clouds; these feature points help the algorithm match more accurately. Some commonly used features include surface normals and curvature information of specific regions. More advanced registration algorithms, such as variants of ICP and non-rigid registration methods, can be used for more precise point cloud matching. These algorithms are generally better suited for matching under rigid body transformations. Finally, evaluation metrics can be used to measure the accuracy of the matching, such as the average distance between point clouds and the number of corresponding points. If the matching results are unsatisfactory, consider optimizing the feature extraction or matching algorithm, and then visualize the matching results to check their accuracy and whether they meet the actual requirements.

[0097] Furthermore, other point clouds in the workpiece to be processed can be determined based on the selected point cloud template and scene point cloud.

[0098] In this implementation, after obtaining the selected point cloud template, all point clouds in the selected point cloud template can be known. Using all point clouds and scene point clouds, other point clouds in the workpiece to be processed that have not been captured in the camera's field of view can be determined.

[0099] The point cloud processing method for non-unilateral workpieces provided in this disclosure involves determining the visible point clouds of each point cloud template within the camera's field of view based on the attribute information of the template points and the corresponding viewpoint information of the camera. Then, based on the scene point cloud of the workpiece to be processed and the visible point clouds corresponding to each template, a point cloud template corresponding to the workpiece is selected from all the workpiece's point cloud templates. Finally, the scene point cloud is fine-matched according to the visible point clouds corresponding to the selected templates. This technical solution utilizes camera viewpoint information and the attribute information of each template point to filter the visible point clouds within the camera's field of view, thereby quickly achieving matching between the scene point cloud and the point cloud templates, increasing the efficiency of fine-matching.

[0100] Based on the above embodiments, the following are three possible solutions for step 11:

[0101] The first type uses the viewpoint information as the viewpoint vector pointing from the camera to the template point, and the attribute information as the normal vector of the template point. Figure 2 A flowchart illustrating the point cloud processing method for non-unilateral workpieces provided in this embodiment of the disclosure. Figure 2 ,like Figure 2 As shown, the implementation of step 11 above may include the following steps:

[0102] in, Figure 3 Schematic diagram of the determination of visible point clouds Figure 1 .

[0103] Step 21: For each template point, determine the angle between the normal vector of each template point and its corresponding viewpoint vector;

[0104] In this step, for each template point in the point cloud template, the angle between the view vector and the normal vector is determined based on the view vector pointing from the camera to the template point and the normal vector of the template point itself.

[0105] like Figure 3 As shown, the point cloud template includes multiple template points, where the dashed line with arrows is the normal vector of the template point, and the solid line with arrows is the view vector of the camera pointing to the template point.

[0106] Taking template point 1 and template point 2 as examples, the included angle corresponding to template point 1 is the angle between the normal vector of template point 1 and the corresponding viewpoint vector, i.e., angle A; the included angle corresponding to template point 2 is the angle between the normal vector of template point 2 and the corresponding viewpoint vector, i.e., angle B.

[0107] For example, angle A is 130° and angle B is 30°.

[0108] Step 22: Take the set of template points where the angle between all normal vectors and the corresponding viewpoint vectors is obtuse as the visible point cloud of the point cloud template in the camera's field of view.

[0109] In this step, all template points with non-obtuse angles are deleted, and the set of all template points with obtuse angles between their normal vectors and the corresponding viewpoint vectors is taken as the visible point cloud under the camera's field of view.

[0110] like Figure 3 As shown, the normal vectors of template points 1, 3, 4, 5, 6, and 7 are at obtuse angles to their respective viewpoint vectors. Therefore, the template points in the visible point cloud are template points 1, 3, 4, 5, 6, and 7.

[0111] The point cloud processing method for non-unilateral workpieces provided in this disclosure determines the angle between the normal vector of each template point and its corresponding viewpoint vector. Then, the set of all template points whose normal vectors form obtuse angles with their corresponding viewpoint vectors is used as the visible point cloud of the point cloud template within the camera's field of view. This technical solution, starting from the angle between the viewpoint vector and the normal vector, quickly achieves the filtering of visible point clouds within the camera's field of view.

[0112] The second method uses viewpoint information as the camera's optical axis direction and attribute information as the vertical component of the template point's normal vector. Figure 4 A flowchart illustrating the point cloud processing method for non-unilateral workpieces provided in this embodiment of the disclosure. Figure 3 ,like Figure 4 As shown, step 11 above may include the following steps:

[0113] in, Figure 5 Schematic diagram of the determination of visible point clouds Figure 2 .

[0114] Step 41: Compare the orientation of the vertical axis component of each template point with the direction of the optical axis;

[0115] In this step, the orientation of the vertical axis component of each template point is compared with the optical axis direction to determine whether the orientation of the vertical axis component of each template point is the same as or opposite to the optical axis direction.

[0116] Step 42: Take the set of template points whose orientation is opposite to the optical axis as the visible point cloud of the point cloud template in the camera's field of view.

[0117] In this step, since the orientation of the vertical component of the normal vector of the points in the camera's field of view is opposite to the direction of the camera's optical axis, the set of template points corresponding to the orientation of all vertical components opposite to the direction of the optical axis can be taken as the visible point cloud in the camera's field of view.

[0118] like Figure 5As shown, the point cloud template includes multiple template points. The long dashed line with arrows represents the normal vector of the template point, the solid line with arrows represents the optical axis direction of the camera, and the dense dashed line with arrows represents the orientation of the vertical component of the normal vector of the template point.

[0119] Taking template point 1 and template point 2 as examples, the orientation of the vertical axis component of template point 1 is opposite to the direction of the optical axis; the orientation of the vertical axis component of template point 2 is the same as the direction of the optical axis.

[0120] like Figure 5 As shown, the vertical axis components of the normal vectors of template points 1, 3, 4, 5, 6, and 7 are oriented opposite to the direction of the optical axis. Therefore, the template points in the point cloud are template points 1, 3, 4, 5, 6, and 7.

[0121] The point cloud processing method for non-unilateral workpieces provided in this disclosure compares the orientation of the vertical axis component of each template point with the optical axis direction, and uses the set of template points whose vertical axis component orientation is opposite to the optical axis direction as the visible point cloud of the point cloud template in the camera's field of view. This technical solution utilizes the optical axis direction to quickly filter template points in the visible point cloud, and because it removes points opposite to the optical axis direction, it can actually retain more template points as the visible point cloud.

[0122] The third type uses viewpoint information as the camera's optical axis direction and attribute information as depth information. Figure 6 A flowchart illustrating the point cloud processing method for non-unilateral workpieces provided in this embodiment of the disclosure. Figure 4 ,like Figure 6 As shown, step 11 above may include the following steps:

[0123] in, Figure 7 Schematic diagram of the determination of visible point clouds Figure 3 .

[0124] Step 61: Based on the optical axis direction, perform orthographic projection on each template point;

[0125] In this step, each template point on each point cloud template is orthographically projected along the optical axis.

[0126] like Figure 7 As shown in the diagram, the schematic diagram includes a point cloud template and a projection plane. The point cloud template includes multiple template points, and each template point is orthographically projected.

[0127] Step 62: Take the set of all template points that do not overlap in orthographic projection as the first part of the visible point cloud of the point cloud template in the camera's field of view;

[0128] In this step, after orthographically projecting each template point, there are two possibilities: some template points are orthographically projected only once, while some points will have overlapping orthographic projections, meaning that the template point will be orthographically projected twice or more.

[0129] At this point, the set of template points that do not overlap with orthographic projections is taken as the first part of the visible point cloud in the camera's field of view.

[0130] like Figure 7 As shown, all template points that do not overlap in orthographic projection are template point 2, template point 3, template point 4, template point 5, and template point 6. Therefore, the first part of the visible point cloud includes: template point 2, template point 3, template point 4, template point 5, and template point 6.

[0131] Step 63: For at least two template points where orthographic projection overlap occurs, obtain the depth information between each template point and the corresponding orthographically projected point.

[0132] In this step, at least two template points that overlap in orthographic projection are used to obtain the depth information between each template point and the corresponding orthographically projected point (which is located on the projection plane) during the orthographic projection process.

[0133] like Figure 7 As shown, all template points that overlap in orthographic projection are template point 1, template point 9, template point 7, and template point 8.

[0134] For example, the depth information of template point 1 is M, and the depth information of template point 9 is N, where M is greater than N.

[0135] Step 64: Take the set of template points with the largest depth information corresponding to the same orthographic projection as the second part of the visible point cloud of the point cloud template in the camera's field of view.

[0136] In this step, the maximum depth information means that a point among at least two template points that have orthographic projection overlap is projected to the projection plane at the maximum distance, which is also the closest point to the vertical plane of the camera's optical axis, and is the visible point in the camera's field of view.

[0137] Furthermore, the set of template points with the greatest depth information is determined as the second part of the visible point cloud under the camera's field of view.

[0138] The second part of the point cloud can be seen to include: template point 1 and template point 7.

[0139] In summary, all the template points in the point cloud are template point 2, template point 3, template point 4, template point 5, template point 6, template point 1, and template point 7.

[0140] The point cloud processing method for non-unilateral workpieces provided in this disclosure performs orthographic projection on each template point based on the optical axis direction. The set of all template points that do not overlap in orthographic projection is used as the first part of the visible point cloud of the point cloud template in the camera's field of view. Then, for at least two template points that overlap in orthographic projection, the depth information between each template point and the corresponding orthographically projected point is obtained. Finally, the set of all template points with the largest depth information corresponding to the same orthographic projection is used as the second part of the visible point cloud of the point cloud template in the camera's field of view. This technical solution uses the template point first encountered during orthographic projection as the template point in the visible point cloud, thereby achieving the deletion of template points that are not visible in the camera's field of view.

[0141] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0142] Figure 8 This is a schematic diagram of the point cloud processing device for non-unilateral workpieces provided in an embodiment of this disclosure.

[0143] like Figure 8 As shown, the point cloud processing device for the non-unilateral workpiece may include:

[0144] The first determining module 81 is used to determine the visible point cloud of each point cloud template in the camera's field of view based on the attribute information of the template points in the point cloud template and the corresponding viewpoint information of the camera, for each point cloud template corresponding to each workpiece. The attribute information is either vector information or depth information obtained by orthographic projection of the template points.

[0145] The second determining module 82 is used to select the point cloud template corresponding to the workpiece to be processed from all the point cloud templates of all workpieces, based on the scene point cloud of the workpiece to be processed and the visible point cloud corresponding to each point cloud template.

[0146] The processing module 83 is used to perform fine matching processing on the scene point cloud based on the visible point cloud corresponding to the selected point cloud template.

[0147] In one or more embodiments, the viewpoint information is a viewpoint vector pointing from the camera to the template point, and the attribute information is the normal vector of the template point;

[0148] Accordingly, the first determining module 81 is specifically used for:

[0149] For each template point, determine the angle between the normal vector of each template point and its corresponding viewpoint vector;

[0150] The set of template points where the angle between all normal vectors and the corresponding viewpoint vectors is obtuse is taken as the visible point cloud of the point cloud template in the camera's field of view.

[0151] In one or more embodiments, the viewpoint information is the optical axis direction of the camera, and the attribute information is the vertical component of the normal vector of the template point;

[0152] Accordingly, the first determining module 81 is specifically used for:

[0153] Compare the orientation of the vertical axis component of each template point with the direction of the optical axis;

[0154] The set of template points whose vertical axis components are oriented opposite to the optical axis is taken as the visible point cloud of the point cloud template in the camera's field of view.

[0155] In one or more embodiments, the viewpoint information is the optical axis direction of the camera, and the attribute information is depth information;

[0156] Accordingly, the first determining module 81 is specifically used for:

[0157] Orthographic projection is performed on each template point based on the optical axis direction;

[0158] The set of all template points that do not overlap in orthographic projection is taken as the first part of the visible point cloud of the point cloud template in the camera's field of view;

[0159] For at least two template points where orthographic projection overlap occurs, obtain the depth information between each template point and the corresponding orthographically projected point.

[0160] The set of template points with the largest depth information corresponding to the same orthographic projection is taken as the second part of the visible point cloud of the point cloud template in the camera's field of view.

[0161] In one or more embodiments, the viewpoint information is a viewpoint vector, which is a vector pointing from the camera to the template point, and the attribute information is vector information, which is the normal vector of the template point.

[0162] Accordingly, before determining the visible point cloud of the point cloud template under the camera's field of view based on the attribute information of each template point in the point cloud template and the corresponding viewpoint information of the camera, the first determining module 81 is also used for:

[0163] Determine whether the angle between the normal vector of the template point and the view vector corresponding to the template point is an acute angle;

[0164] If there exists a template point whose angle between the normal vector and the corresponding viewpoint vector is acute, then the visible point cloud of the point cloud template in the camera's field of view is determined based on the attribute information of the template point in the point cloud template and the corresponding viewpoint information of the camera.

[0165] In one or more embodiments, the first determining module 81 is further configured to:

[0166] Based on the selected point cloud template and scene point cloud, determine the other point clouds in the workpiece to be processed.

[0167] The point cloud processing apparatus for non-unilateral workpieces provided in this disclosure can be used to execute the point cloud processing method for non-unilateral workpieces in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0168] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented via processing element calls in software, while others are implemented in hardware. Additionally, these modules can be fully or partially integrated together, or implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0169] Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this disclosure, such as... Figure 9 As shown, the electronic device may include: a processor 91, a memory 92, and computer program instructions stored in the memory 92 and executable on the processor 91, wherein the processor 91 executes the computer program instructions to implement the method provided in any of the foregoing embodiments.

[0170] Optionally, the various components of the electronic device can be connected via a system bus.

[0171] The memory 92 can be a separate memory unit or a memory unit integrated into the processor 91. The number of processors 91 can be one or more.

[0172] It should be understood that the processor 91 can be a Central Processing Unit (CPU), or other general-purpose processors 91, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor 91 can be a microprocessor 91, or any conventional processor 91. The steps of the method disclosed in this disclosure can be directly manifested as being executed by the hardware processor 91, or being executed by a combination of hardware and software modules within the processor 91.

[0173] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Memory 92 may include random access memory (RAM) 92, and may also include non-volatile memory (NVM) 92, such as at least one disk storage device 92.

[0174] All or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory 92. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory 92 (storage medium) includes: read-only memory 92 (ROM), RAM, flash memory 92, hard disk, solid-state hard disk, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0175] The electronic device provided in this disclosure can be used to execute the point cloud processing method for non-unilateral workpieces provided in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0176] This disclosure provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the aforementioned point cloud processing method for non-unilateral workpieces.

[0177] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0178] Optionally, a readable storage medium can be coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Alternatively, the readable storage medium can be an integral part of the processor. Both the processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components within the device.

[0179] This disclosure also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the above-described point cloud processing method for non-unilateral workpieces.

[0180] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A point cloud processing method for a non-unilateral workpiece, characterized in that, include: For each workpiece, a point cloud template is provided. Based on the attribute information of the template points in the point cloud template and the corresponding viewpoint information of the camera, the visible point cloud of each point cloud template under the camera's field of view is determined. When the viewpoint information is the viewpoint vector pointing from the camera to the template point, the attribute information is the normal vector of the template point. When the viewpoint information is the optical axis direction of the camera, the attribute information is the vertical component of the normal vector of the template point; or, when the viewpoint information is the optical axis direction of the camera, the attribute information is depth information. Based on the scene point cloud of the workpiece to be processed and the visible point cloud corresponding to each point cloud template, select the point cloud template corresponding to the workpiece to be processed from the point cloud templates of all workpieces. Based on the visible point cloud corresponding to the selected point cloud template, the scene point cloud is subjected to fine matching processing.

2. The method according to claim 1, characterized in that, The viewpoint information is the viewpoint vector pointing from the camera to the template point, and the attribute information is the normal vector of the template point; Accordingly, determining the visible point cloud of each point cloud template under the camera's field of view based on the attribute information of the template points in the point cloud template and the corresponding viewpoint information of the camera includes: For each template point, determine the angle between the normal vector of the template point and its corresponding viewpoint vector; The set of template points where the angle between all the normal vectors and the corresponding viewpoint vectors is obtuse is taken as the visible point cloud of the point cloud template in the camera's field of view.

3. The method according to claim 1, characterized in that, The viewpoint information is the optical axis direction of the camera, and the attribute information is the vertical component of the normal vector of the template point; Accordingly, determining the visible point cloud of each point cloud template under the camera's field of view based on the attribute information of the template points in the point cloud template and the corresponding viewpoint information of the camera includes: Compare the orientation of the vertical axis component of each template point with the direction of the optical axis; The set of template points whose orientation is opposite to that of the optical axis is taken as the visible point cloud of the point cloud template in the camera's field of view.

4. The method according to claim 1, characterized in that, The viewpoint information is the optical axis direction of the camera, and the attribute information is the depth information; Accordingly, determining the visible point cloud of each point cloud template under the camera's field of view based on the attribute information of the template points in the point cloud template and the corresponding viewpoint information of the camera includes: Based on the optical axis direction, orthographic projection is performed on each template point; The set of all template points that do not overlap in orthographic projection is taken as the first part of the visible point cloud of the point cloud template in the camera's field of view; For at least two template points where orthographic projection overlap occurs, obtain the depth information between each template point and the corresponding orthographically projected point. The set of all template points that have the largest depth information when corresponding to the same orthographic projection is taken as the second part of the visible point cloud of the point cloud template in the camera's field of view.

5. The method according to any one of claims 1-4, characterized in that, The step of determining the visible point cloud of each point cloud template under the camera's field of view based on the attribute information of the template points in the point cloud template and the corresponding viewpoint information of the camera includes: Determine whether the angle between the normal vector of the template point and the viewpoint vector corresponding to the template point is an acute angle; If there exists a template point where the angle between the normal vector and the corresponding viewpoint vector is acute, then the visible point cloud of the point cloud template in the camera's field of view is determined based on the attribute information of the template point in the point cloud template and the corresponding viewpoint information of the camera.

6. The method according to any one of claims 1-4, characterized in that, The method further includes: Based on the selected point cloud template and the scene point cloud, other point clouds in the workpiece to be processed are determined.

7. A point cloud processing device for a non-unilateral workpiece, characterized in that, include: The first determining module is used to determine the visible point cloud of each point cloud template in the field of view of the camera, based on the attribute information of the template points in the point cloud template and the corresponding viewpoint information of the camera, for each point cloud template corresponding to each workpiece; when the viewpoint information is the viewpoint vector pointing from the camera to the template point, the attribute information is the normal vector of the template point. When the viewpoint information is the optical axis direction of the camera, the attribute information is the vertical component of the normal vector of the template point; or, when the viewpoint information is the optical axis direction of the camera, the attribute information is depth information. The second determining module is used to select the point cloud template corresponding to the workpiece to be processed from the point cloud templates of all workpieces, based on the scene point cloud of the workpiece to be processed and the visible point cloud corresponding to each point cloud template. The processing module is used to perform fine matching processing on the scene point cloud based on the visible point cloud corresponding to the selected point cloud template.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-6.

Images