A spatial plane rectangular pose estimation method, device, chip and storage medium

Abstract

This application discloses a method, apparatus, chip, and storage medium for spatial planar rectangle attitude estimation. The method includes: acquiring a 3D point cloud of a target rectangle; performing planar fitting on the 3D point cloud to extract the target plane and interior point set of the target rectangle; acquiring the gradient of each interior point in the interior point set to determine the edge points of the target rectangle; determining the projection distance between the edge points and a reference point along their gradient directions; determining the geometric parameters of the target rectangle based on the gradient directions and projection distances corresponding to the edge points; and determining the attitude parameters of the target rectangle based on the geometric parameters and the normal vector of the target plane. Based on the method of this application, by performing planar fitting on the 3D point cloud of the target rectangle, extracting edge points based on the interior point set of the target plane, constructing joint features of direction and distance, and then solving for the geometric parameters of the rectangle, the complexity of high-dimensional fitting in 3D space can be reduced, the effective distinction of each side of the target rectangle can be achieved, and efficient and accurate estimation of attitude parameters can be realized.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a spatial planar rectangular pose estimation method, apparatus, chip, and storage medium. Background Technology

[0002] In industrial manufacturing scenarios, robotic arms often need to grasp rectangular parts (such as circuit boards, metal plates, and packaging boxes). Generally, attitude estimation of the rectangular parts is required for spatial positioning to facilitate grasping. However, traditional methods struggle to achieve efficient and accurate attitude estimation. Summary of the Invention

[0003] This application provides a spatial planar rectangle attitude estimation method, device, chip, and storage medium. Based on the method described in this application, the method performs planar fitting on the three-dimensional point cloud corresponding to the target rectangle, extracts edge points based on the set of interior points of the target plane, constructs joint features of direction and distance, and then completes the solution of the rectangle's geometric parameters. This transforms the rectangular target in the spatial point cloud into a two-dimensional local geometric problem for processing, which can reduce the solution complexity of directly performing high-dimensional fitting in three-dimensional space, effectively distinguish the sides of the target rectangle, and achieve efficient and accurate estimation of attitude parameters.

[0004] In a first aspect, this application provides a spatial planar rectangle attitude estimation method, which includes: acquiring a three-dimensional point cloud corresponding to a target rectangle; performing planar fitting on the three-dimensional point cloud to extract the target plane where the target rectangle is located and a set of interior points belonging to the target plane in the three-dimensional point cloud; acquiring the gradient of each interior point in the set of interior points to determine the edge points of the target rectangle; for each edge point, determining the projection distance between the edge point and a reference point in the target plane in its gradient direction; clustering the edge points based on the gradient direction and projection distance corresponding to the edge points to determine the geometric parameters of the target rectangle, wherein the geometric parameters include at least one of geometric center, length, width, and in-plane rotation angle; and determining the attitude parameters of the target rectangle based on the geometric parameters and the normal vector of the target plane.

[0005] In some implementations, edge points are clustered based on the gradient direction and projection distance corresponding to the edge points to determine the geometric parameters of the target rectangle. This includes: normalizing the gradient corresponding to the edge points to obtain normalized gradients; clustering the edge points based on the normalized gradients and projection distances to obtain the edge groups corresponding to the four sides of the target rectangle; and determining the geometric parameters of the target rectangle in a local coordinate system based on the edge groups.

[0006] In some implementations, edge points are clustered based on their normalized gradients and projected distances. This includes: normalizing the normalized gradients of edge points based on the radial vectors between the edge points and reference points to obtain processed normalized gradients, such that the processed normalized gradients and the radial vectors satisfy a preset directional relationship. The normalization process includes maintaining the direction of the normalized gradients unchanged or reversing the normalized gradients; and clustering the edge points based on their processed normalized gradients and projected distances.

[0007] In some implementations, edge points are clustered based on their normalized gradients and projected distances. This includes: constructing multidimensional feature vectors corresponding to the edge points, where the multidimensional feature vectors are composed of the coordinate components of the normalized gradients of the edge points and the projected distances; and clustering the edge points based on the multidimensional feature vectors.

[0008] In some implementations, the geometric parameters of the target rectangle are determined in the local coordinate system based on the edge group, including: establishing a system of equations related to the four sides of the target rectangle in the local coordinate system according to the geometric constraints of the rectangle based on the edge group; solving the system of equations to obtain the four corner points of the target rectangle; and determining the geometric parameters of the target rectangle based on the four corner points.

[0009] In some implementations, obtaining the gradient of each interior point in the set of interior points includes: projecting all interior points in the set onto a local coordinate system to obtain the two-dimensional coordinates of each interior point, wherein the local coordinate system is established based on the target plane; and obtaining the gradient of each interior point based on the two-dimensional coordinates of each interior point.

[0010] In some implementations, the gradient of each interior point is obtained based on its two-dimensional coordinates, including: determining the projection resolution based on the statistical value of the minimum nearest neighbor distance of each point in the three-dimensional point cloud; determining the two-dimensional floating-point coordinates of each interior point based on its two-dimensional coordinates and the projection resolution; and determining the gradient of each interior point based on its two-dimensional floating-point coordinates.

[0011] In some implementations, the gradient of each interior point is determined based on its two-dimensional floating-point coordinates, including: generating two-dimensional integer coordinates of each interior point based on its two-dimensional floating-point coordinates; and determining the gradient of each interior point based on its two-dimensional integer coordinates.

[0012] In some implementations, reference points include: the geometric center of all edge points; the local center of some edge points; and reference points determined based on historical poses.

[0013] In some implementations, the method further includes: obtaining a subset of the 3D point cloud from the 3D point cloud based on the pose parameters of the target rectangle and a preset distance threshold; and performing pose estimation on the target rectangle based on the subset of the 3D point cloud to update the pose parameters of the target rectangle.

[0014] Secondly, this application provides an apparatus, which can be a standalone device, such as a terminal device or a server, or a processing device within a device, such as a processing system in a terminal device or server. The apparatus can also be a chip system. This apparatus can execute the spatial planar rectangular pose estimation method described in the first aspect. The functionality of this apparatus can be implemented in hardware or by executing corresponding software. The hardware or software includes one or more units corresponding to the aforementioned functions. These units can be software and / or hardware. The operations performed by this apparatus and its beneficial effects are described in the first aspect and its beneficial effects above; repetitions will not be repeated.

[0015] Thirdly, an apparatus is provided, including a processor. The processor is coupled to a memory and can be used to execute instructions or data in the memory to implement the spatial planar rectangular pose estimation method as described in the first aspect. Optionally, the apparatus further includes a memory. Optionally, the apparatus also includes a communication interface, to which the processor is coupled.

[0016] In one implementation, the communication interface can be a transceiver, or an input / output interface.

[0017] In one implementation, the device can be a chip configured within a device. When the device is a chip configured within a device, the communication interface can be an input / output interface.

[0018] Fourthly, this application provides an apparatus comprising a processor and a memory for storing computer execution instructions; the processor is configured to execute the computer execution instructions stored in the memory to cause the apparatus to perform the spatial planar rectangular pose estimation method as described in the first aspect.

[0019] Optionally, there may be one or more processors and one or more memories.

[0020] Fifthly, this application provides an apparatus comprising a processor, a memory, and a transceiver, the transceiver being used to receive or transmit signals; the memory being used to store a computer program; and the processor being used to invoke the computer program from the memory to execute the spatial planar rectangular attitude estimation method as described in the first aspect.

[0021] In a sixth aspect, this application provides an apparatus including a processor and an interface circuit for receiving computer execution instructions and transmitting them to the processor; the processor executes the computer execution instructions to perform the spatial planar rectangular pose estimation method as described in the first aspect.

[0022] In a seventh aspect, a processor is provided, comprising: an input circuit, an output circuit, and a processing circuit. The processing circuit is used to receive signals through the input circuit and to transmit signals through the output circuit, causing the processor to execute the spatial planar rectangular pose estimation method as described in the first aspect.

[0023] In specific implementation, the processor can be one or more chips, the input circuit can be input pins, the output circuit can be output pins, and the processing circuit can be transistors, gate circuits, flip-flops, and various logic circuits. The input signal received by the input circuit can be received and input by, for example, but not limited to, a receiver, and the signal output by the output circuit can be, for example, but not limited to, output to a transmitter and transmitted and output by the transmitter. Furthermore, the input circuit and the output circuit can be the same circuit, which is used as the input circuit and output circuit at different times. This application does not limit the specific implementation of the processor and various circuits.

[0024] Eighthly, this application provides a computer program product comprising: a computer program (also referred to as code or instructions) that, when run, causes a computer to perform the spatial planar rectangular pose estimation method as described in the first aspect.

[0025] In a ninth aspect, this application provides a storage medium for storing computer-executable instructions that, when executed, cause the computer to perform the spatial planar rectangular pose estimation method as described in the first aspect.

[0026] In a tenth aspect, this application provides an apparatus comprising functions or units for performing any of the methods in the first aspect.

[0027] Eleventhly, embodiments of this application provide a chip system including one or more processors for calling and executing instructions stored in memory, causing the methods in any of the above aspects or possible implementations to be executed. The chip system may be composed of chips or may include chips and other discrete devices.

[0028] The chip system may include input circuits or interfaces for transmitting information or data, and output circuits or interfaces for receiving information or data.

[0029] This application transforms the rectangular target in the spatial point cloud into a two-dimensional local geometric problem by performing planar fitting on the three-dimensional point cloud corresponding to the target rectangle, extracting edge points based on the set of interior points of the target plane, constructing joint features of direction and distance, and then solving the geometric parameters of the rectangle. This reduces the solution complexity of directly performing high-dimensional fitting in three-dimensional space, effectively distinguishes the sides of the target rectangle, and achieves efficient and accurate estimation of attitude parameters. Attached Figure Description

[0030] Figure 1 A schematic diagram of a system architecture provided for an embodiment of this application; Figure 2 A schematic flowchart illustrating a spatial planar rectangular pose estimation method provided in an embodiment of this application; Figure 3 A schematic diagram of the normalized gradient of the edge points of each side of the rectangle provided in the embodiments of this application; Figure 4 A schematic diagram illustrating an example of a reference point provided in an embodiment of this application; Figure 5 A schematic diagram of the gradient direction and projection distance provided in an embodiment of this application; Figure 6 A schematic diagram of normalized gradient uniformity processing provided in an embodiment of this application; Figure 7 A flowchart illustrating another spatial planar rectangular pose estimation method provided in this application embodiment; Figure 8 A schematic diagram of the structure of an apparatus provided in an embodiment of this application; Figure 9 A schematic diagram of another device provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0031] The terms "first" and "second," etc., used in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0032] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0033] In this application, "at least one (item)" means one or more, "more than one" means two or more, "at least two (items)" means two or three or more, and "and / or" is used to describe the correspondence between related objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0034] The terms "comprising" and "having," and any variations thereof, used in the following description of embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. It should be noted that in embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any method or design described as "exemplary" or "for example" in embodiments of this application should not be construed as preferred or advantageous over other methods or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0035] To better understand the embodiments of this application, the architecture involved in the embodiments of this application will be described first below: Figure 1 This is a schematic diagram of a system architecture provided in an embodiment of this application.

[0036] like Figure 1As shown, the system architecture 100 may include a first terminal device 101, a second terminal device 102, an image acquisition device 103, a network 104, a server 105, a robotic arm 106, and a target rectangular object 107. The image acquisition device 103 can be used to acquire the 3D point cloud of the target rectangular object 107, and the network 104 can be used to provide a communication link between the first terminal device 101, the second terminal device 102, the image acquisition device 103, the server 105, and the robotic arm 106. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0037] The first terminal device 101 and the second terminal device 102 can interact with the server 105 via the network 104 to receive or send messages, etc. The image acquisition device 103 can interact with the server 105 or the first terminal device 101 and the second terminal device 102 via the network 104. The first terminal device 101 and the second terminal device 102 can be various electronic devices with displays and processing capabilities, including but not limited to smartphones, tablets, laptops, desktop computers, wearable devices, virtual reality devices, smart home devices, etc.

[0038] Server 105 can be a server that provides various services. For example, it can be a backend management server that supports the operations performed by the first terminal device 101, the second terminal device 102, or the image acquisition device 103, or a backend service manager that supports the movement and positioning operations of the robotic arm 106. The backend management server can analyze and process the data received from the first terminal device 101, the second terminal device 102, or the image acquisition device 103, and send the processing results to the robotic arm 106 to control the robotic arm 106 to perform actions such as movement or grasping.

[0039] The robotic arm 106 can be used to grasp objects. The target rectangular object 107 can be the rectangular face of one or more regularly arranged or randomly arranged objects, or it can be a rectangular object.

[0040] It should be understood that Figure 1 The number of terminal devices, image acquisition devices, networks, servers, and robotic arms shown is merely illustrative. Server 105 can be a single physical server or a combination of multiple servers. Depending on actual needs, it can have any number of terminal devices, image acquisition devices, networks, servers, and robotic arms.

[0041] Regarding pose estimation for spatial planar rectangles, traditional methods, such as those based on RGB images and depth mapping, provide incomplete pose estimations, making it difficult to extract complete spatial geometric information and output accurate pose parameters. Another example is the degradation scheme based on cuboid fitting, which involves a huge sampling space and computational redundancy. Therefore, existing techniques struggle to achieve efficient and accurate pose estimation.

[0042] Figure 2 This is a flowchart illustrating a spatial planar rectangular pose estimation method provided in an embodiment of this application. The execution entity of this method can be the aforementioned terminal device or server. Alternatively, Figure 2 The method shown can be executed by a chip in a terminal device or server; this application does not limit this. For ease of description, Figure 2 The following explanation uses the server as the execution subject of the method.

[0043] S201. Obtain the 3D point cloud corresponding to the target rectangle.

[0044] The server can obtain the 3D point cloud corresponding to the target rectangle. The target rectangle can be a rectangular workpiece, rectangular marking line, rectangular plate, or other target object with a rectangular outline located on any spatial plane; this application does not limit this.

[0045] The server can, for example, Figure 1 Image acquisition devices, or depth cameras, structured light devices, lidar, or other sensors capable of acquiring three-dimensional spatial information, are used to obtain the three-dimensional point cloud corresponding to the target rectangle.

[0046] The acquired 3D point cloud can reflect the spatial geometric distribution of the target rectangle, providing the original data basis for subsequent plane extraction, edge recognition and pose estimation.

[0047] S202. Perform plane fitting on the 3D point cloud to extract the target plane containing the target rectangle and the set of interior points in the 3D point cloud that belong to the target plane.

[0048] For the acquired 3D point cloud, the server can use a plane fitting method to extract the target plane containing the target rectangle. For example, plane fitting can be implemented based on RANSAC, least squares, or any other suitable robust plane estimation method to extract the target plane corresponding to the target rectangle from the 3D point cloud containing background points, outliers, and noise points, and then based on a wide plane distance threshold (e.g., denoted as...). A distance of 0.01m can be used to extract the set of points belonging to the target plane. Points whose distance from the target plane is within the plane distance threshold range can be considered to belong to the target plane. These points can be called interior points of the target plane, and the set of interior points can be simply referred to as the interior point set.

[0049] By fitting a plane and obtaining a set of interior points of the target plane, most non-target points (e.g., background points, outliers, and noise points) can be eliminated, allowing subsequent processing to focus on the local planar region where the target rectangle is located, reducing external interference and improving the accuracy of subsequent edge extraction and the stability of geometric solutions.

[0050] S203. Obtain the gradient of each interior point in the set of interior points to determine the edge points of the target rectangle.

[0051] The server can obtain the gradient information corresponding to each inlier in the inlier set and determine the edge points of the target rectangle based on the gradient information. Since the edge region corresponds to significant changes in the graphic image, its gradient value is usually greater than that of the outer region and the inner region of the graphic. Therefore, the edge points located near the edge of the target rectangle can be identified from the inlier set based on the gradient information. For example, inlier points with gradient magnitudes greater than a preset threshold can be identified as edge points of the target rectangle, and this provides boundary features for the subsequent determination of the rectangle's geometric parameters.

[0052] In some embodiments, obtaining the gradient of each interior point in the set of interior points may include: projecting all interior points in the set onto a local coordinate system to obtain the two-dimensional coordinates of each interior point, wherein the local coordinate system is established based on the target plane; and obtaining the gradient of each interior point based on the two-dimensional coordinates of each interior point.

[0053] The server can establish a local coordinate system based on the fitted target plane. This local coordinate system can include a first coordinate axis and a second coordinate axis located within the target plane, as well as a third coordinate axis perpendicular to the target plane. For example, the server can use the normal vector of the target plane as the third coordinate axis, and construct the first and second coordinate axes to form an orthogonal coordinate system based on the third coordinate axis.

[0054] Based on the constructed local coordinate system, the server can project all interior points in the interior point set onto the fitted target plane. Since the projected point set has the same normal axis (i.e., the aforementioned third coordinate axis) coordinates, the server can use the coordinates corresponding to the first and second coordinate axes as the two-dimensional coordinates of each interior point for subsequent gradient processing.

[0055] Based on this implementation, by projecting the set of interior points onto a local coordinate system established based on the target plane, and obtaining the gradient of each interior point in the two-dimensional local coordinate system, it is beneficial to use mature two-dimensional image gradient algorithms to obtain gradients. Compared with performing gradient estimation directly on the three-dimensional point cloud, it has the advantages of high computational efficiency and stable results. At the same time, it can provide a unified coordinate basis for subsequent rectangular geometric parameter solving in the local coordinate system.

[0056] In some embodiments, obtaining the gradient of each interior point based on its two-dimensional coordinates may include: determining the projection resolution based on the statistical value of the minimum nearest neighbor distance of each point in the three-dimensional point cloud; determining the two-dimensional floating-point coordinates of each interior point based on its two-dimensional coordinates and the projection resolution; and determining the gradient of each interior point based on its two-dimensional floating-point coordinates.

[0057] For any target rectangle and / or a 3D point cloud of a target rectangle obtained in any suitable manner, the server can adaptively determine the projection resolution required for projection onto the local coordinate system.

[0058] Alternatively, the server can use the voxel size of the point cloud voxel mesh as the projection resolution, or estimate the projection resolution based on the intrinsic parameters of the image acquisition device and the depth noise model.

[0059] Optionally, the server can adaptively determine the projection resolution based on statistical values ​​of the minimum nearest neighbor distances between points in the 3D point cloud.

[0060] Proximity distance can refer to the distance between a point in a 3D point cloud and its neighboring points (e.g., points within a preset range). For any single point in a 3D point cloud, its minimum proximity distance is the minimum value of the distance between that point and its neighboring points.

[0061] Optionally, the distance between points can be the Euclidean distance between points. That is, the minimum nearest neighbor distance can be the minimum Euclidean distance between a point and its surrounding nearest points.

[0062] Optionally, the statistical value of the minimum neighbor distance can be any one of the arithmetic mean, geometric mean, or weighted average of the minimum neighbor distances.

[0063] Taking Euclidean distance and arithmetic mean as examples, the server can analyze point clouds. For each point, search for its nearest neighbor, calculate the Euclidean distance between points, and use the arithmetic mean of all minimum Euclidean distances as the adaptive projection resolution. , The following formula can be satisfied:

[0064] After projecting the interior points of the target plane onto the local coordinate system, the server can obtain the two-dimensional floating-point coordinates of each interior point based on the two-dimensional coordinates of each interior point in the local coordinate system and the projection resolution, and determine the gradient of each interior point based on the two-dimensional floating-point coordinates of each interior point.

[0065] For example, by projecting all interior points from the interior point set onto the local coordinate system, we can obtain the two-dimensional coordinates of all interior points in the local coordinate system, which can be denoted as follows: The server can determine and Their respective minimum and maximum values ​​can be denoted as, for example, as... , , , For each interior point, the server can... and Calculate the two-dimensional floating-point coordinates of the interior points based on their respective minimum values. . , The following formula can be satisfied:

[0066] Among them, the server can It is retained as a double type and not rounded.

[0067] Optionally, in calculating two-dimensional floating-point coordinates Previously, the server could also extend the boundary of the calculated floating-point coordinates outward by one or more resolution-sized margins to prevent edge points from being truncated.

[0068] For example, servers can be scaled out. The margin, that is, for , , , The assignment is as follows:

[0069] Expand outward After setting the margins, the server can utilize the methods listed above. , The formula calculates the two-dimensional floating-point coordinates of each interior point. .

[0070] Based on this implementation, the projection resolution is determined by statistically analyzing the minimum nearest neighbor distance of each point in the 3D point cloud. This enables adaptive calculation of the projection resolution, allowing the algorithm to automatically adapt to point cloud inputs with different sampling densities without requiring manual pre-setting of this parameter. This avoids the problems of over-quantization leading to image discontinuities in sparse point clouds and computational redundancy in dense point clouds caused by fixed resolution parameters. Furthermore, by generating 2D floating-point coordinates for each interior point based on the determined projection resolution and determining the gradient of each interior point based on these coordinates, sub-pixel-level coordinate accuracy can be preserved in gradient calculation and subsequent rectangle fitting. This avoids the quantization errors introduced when rounding and mapping 3D point cloud coordinates to integer pixel grids, which helps improve the accuracy of the final rectangle's geometric and pose parameters.

[0071] In some implementations, the gradient of each interior point is determined based on its two-dimensional floating-point coordinates. This can include: generating two-dimensional integer coordinates for each interior point based on its two-dimensional floating-point coordinates; and determining the gradient of each interior point based on its two-dimensional integer coordinates.

[0072] When determining the gradient of each interior point based on its two-dimensional floating-point coordinates, the server can first generate the corresponding two-dimensional integer coordinates of each interior point based on the two-dimensional floating-point coordinates; and then determine the gradient of each interior point based on its two-dimensional integer coordinates.

[0073] For example, the server retrieves the interior point. Two-dimensional floating-point coordinates Then, an integer mesh M can be created, where the width of mesh M is... ,high The dimensions can be as follows:

[0074] in, , They are respectively maximum value The maximum value, This indicates rounding up to the nearest integer.

[0075] Create an integer mesh After that, the server can process each internal point. Calculate its two-dimensional floating-point coordinates Corresponding two-dimensional integer coordinates ,in, , They can be respectively , An integer value obtained by any of the appropriate methods of rounding up, rounding down, or rounding down.

[0076] Understandable, interior point Its two-dimensional floating-point coordinates and its two-dimensional integer coordinates There can be a corresponding relationship between them, and the server can maintain internal points. Its two-dimensional floating-point coordinates and its two-dimensional integer coordinates A mapping table between the three or between any two is used to store all floating-point coordinates falling within an integer grid cell and their corresponding projected coordinates in the local coordinate system or their corresponding original 3D point indices.

[0077] Optionally, the server can target internal points. Two-dimensional floating-point coordinates Integer mesh after rounding (That is, the pixel) is assigned a value of 1, which sets it to 1. and for each pixel Calculate its corresponding gradient to reduce computational complexity.

[0078] Alternatively, the server can perform morphological closing operations before gradient calculation, for example, on the mesh. implement The closing operation of the structuring element (dilation followed by erosion) is used to fill local voids that may be caused by the sparse point cloud and connect adjacent broken areas, making the interior of the rectangle continuous, thereby improving the edge continuity obtained by subsequent gradient calculation.

[0079] Alternatively, the server can utilize the Sobel or Scharr operator, based on the values ​​corresponding to the integer mesh. Calculate interior points The corresponding gradient.

[0080] Optionally, the server can calculate pixels. The central difference gradient of the interior point is used as the interior point. corresponding gradient , where gradient The amount and They can be satisfied separately:

[0081] Among them, gradient magnitude The normalized gradient can be expressed as:

[0082] It is understood that a normalized gradient can be used to represent the direction of a gradient. In this application, in some cases, as is well known to those skilled in the art, a normalized gradient can have the same meaning as the gradient direction.

[0083] The server can retain gradient magnitudes greater than a preset threshold (e.g., (like The pixels of the first edge pixel are used as edge pixels, and a mapping table is used to obtain the set of floating-point coordinates corresponding to each edge pixel. Each floating-point coordinate is then used as an edge point. And record its gradient direction. In some examples, the server can output a set of edge points. and edge point sets in 3D point clouds The corresponding boundary point cloud.

[0084] Based on this implementation, by generating two-dimensional integer coordinates for each interior point based on two-dimensional floating-point coordinates, and determining the gradient of each interior point based on these two-dimensional integer coordinates, the gradient calculation operation can be transferred to a regular grid composed of integer coordinates while preserving the sub-pixel precision information carried by the two-dimensional floating-point coordinates. The regular integer coordinate grid facilitates the application of efficient gradient operators such as central difference and morphological operations. Compared to directly estimating the gradient from floating-point coordinates, the computational speed is significantly improved, while also ensuring computational stability. This design, through the decoupling of floating-point and integer coordinates, achieves an effective balance between computational accuracy and efficiency.

[0085] S204. For each edge point, determine the projected distance between the edge point and the reference point in the target plane along its gradient direction.

[0086] According to the definition of gradient (for example, see the formula described above for the central difference gradient), the gradient of the edge points of the target rectangle on the target plane (excluding the four corner vertices) is generally perpendicular to the edge where the edge point is located. In other words, the gradient direction of the edge point points to the normal direction of the edge where the edge point is located. Alternatively, the normalized gradient of the edge point can be the normal vector of the edge where the edge point is located.

[0087] See Figure 3 , Figure 3 This is a schematic diagram of the normalized gradients of the edge points of each side of the rectangle provided in an embodiment of this application. For example... Figure 3 As shown, the gradient directions of the edge points (except for the four corner vertices) on the four sides of rectangle 300 are all perpendicular to their respective sides.

[0088] Based on this, for each edge point, the server can determine the projected distance between the edge point and the reference point in the target plane along the gradient direction according to the gradient direction corresponding to the edge point. This projected distance is the distance from the reference point in the target plane to the edge where the edge point is located.

[0089] Since the reference point in the target plane corresponds to four distances from the four sides of the target rectangle, the projected distances corresponding to different edge points can serve as important features for distinguishing different boundaries, providing a basis for the classification of rectangle boundaries.

[0090] In some embodiments, the reference point can be any suitable reference point within the target plane other than the edge points of the target rectangle.

[0091] Optionally, the reference point can be a benchmark point located outside the target rectangle within the target plane.

[0092] Optionally, the reference point can be a benchmark point located inside the target rectangle within the target plane.

[0093] For example, the reference point can be the geometric center of all edge points of the target rectangle, and the geometric center can be denoted as... It satisfies the following formula:

[0094] For example, the reference point can be the local center of the edge points of the target rectangle (e.g., randomly sampled edge points).

[0095] For example, the reference point can be the center of all interior points in the set of interior points of the target rectangle.

[0096] For example, the reference point can be any benchmark point set or selected inside the target rectangle.

[0097] Optionally, the reference point can be determined based on historical poses. For example, in a continuous pose estimation scenario, the server can use the center of the target rectangle from historical pose estimations as the reference point.

[0098] See Figure 4 , Figure 4 A schematic diagram illustrating an example of a reference point provided in an embodiment of this application. Figure 4 In the diagram, 'a' shows an example where the first reference point 415 is located outside the target rectangle; Figure 4 b in the figure shows an example where the second reference point 455 is located inside the target rectangle, for example, it could be a local center, an interior center, or an interior point determined based on historical pose estimation; Figure 4 The 'c' in the diagram illustrates an example where the third reference point 495 is the geometric center of the target rectangle. , , , These represent the distances from the reference point to each side of the rectangle in each example.

[0099] Since the projected distance between an edge point and a reference point along its gradient direction is the same as the distance between the edge point and the reference point, the distance from the reference point to each side of the rectangle can provide a distance reference benchmark for all edge points. This makes the projected distances corresponding to the same side have clustering consistency and makes the projected distances corresponding to different sides separable. Therefore, the reference point can provide a basis for classifying the rectangle boundary.

[0100] S205. Cluster the edge points based on the gradient direction and projection distance corresponding to the edge points to determine the geometric parameters of the target rectangle.

[0101] After determining the projection distance, the server can comprehensively analyze and process the edge points by utilizing the gradient direction and projection distance corresponding to each edge point. This divides the edge points into edge groups corresponding to each side of the target rectangle, thereby determining the geometric parameters of the target rectangle. This is because there may be cases where edge points on different sides have the same projection distance to the reference point, for example, ... Figure 4 As shown in c, , Therefore, by comprehensively utilizing the gradient direction and projection distance corresponding to the edge points for edge group clustering, the accuracy and stability of edge group clustering can be improved, so as to further determine the geometric parameters of the target rectangle based on the edge group. The geometric parameters may include one or more of the following: the four vertices (or corner points) of the rectangle, the geometric center of the rectangle, the length, the width, and the rotation angle in the plane.

[0102] See Figure 5 , Figure 5 This is a schematic diagram of the gradient direction and projection distance provided in an embodiment of this application.

[0103] like Figure 5 As shown, the gradients of the edge points on edges P, Q, R, and S of the rectangle all point inwards. The gradient direction of the edge point on edge P is downwards, the gradient direction of the edge point on edge Q is to the left, the gradient direction of the edge point on edge R is upwards, and the gradient direction of the edge point on edge S is to the right. By comprehensively utilizing the gradient direction and projected distance of the edge points, edge group clustering can be performed on the edge points along two dimensions: gradient direction and projected distance, even if a reference point exists. The distances to different edges are equal (e.g., Figure 5 In the middle, reference point Distance to edge P Distance to edge R Even when the edges are equal, the edge groups can be clustered accurately and stably, so that the server can use the accurately classified edge groups to determine the geometric parameters of the target rectangle.

[0104] In some embodiments, the angle can be doubled directly using the double-angle formula. and The gradient is obtained from the quadratic term. The direction. For example, the direction angle is denoted as... ,but It can satisfy:

[0105] Based on this implementation method, it is possible to avoid [the following issues]. Direct division can eliminate The singularity of, and compress the direction to The range is suitable for edge-symmetric scenarios.

[0106] In some embodiments, the gradient direction can be represented using complex numbers.

[0107] For example, it can be and Combination of complex numbers At this point, in the complex domain, direction information can be encoded in the phase. It can be extracted in the following ways. :

[0108] Furthermore, the direction-weighted average can be expressed as The direction is Directional similarity can be expressed as .

[0109] Based on this implementation method, the angle modulus can be automatically processed. The periodicity and directional average will not result in the error that the average of 0° and 360° is 180°.

[0110] In some embodiments, the gradient direction can be determined based on the gradient structure tensor.

[0111] For example, it can be used and Construct a structure tensor matrix and obtain robust principal directions through eigenvalue decomposition.

[0112] Step 1. Construct the structure tensor pixel by pixel:

[0113] Step 2. Perform neighborhood Gaussian weighted smoothing (e.g., let the window be...). The weight is ):

[0114] in, , * represents convolution operation.

[0115] Step 3. Eigenvalue decomposition:

[0116] Step 4. Extract the principal gradient direction (corresponding to) (Eigenvector direction)

[0117] Or equivalent:

[0118] Based on this implementation method, Gaussian smoothing can make the direction estimation more robust and resistant to noise.

[0119] In some embodiments, the gradient direction can also be represented using a normalized gradient. Using a normalized gradient to represent the gradient direction eliminates the need to calculate angles; the direction can be directly represented using the two components of the direction unit vector. This allows all subsequent direction operations (e.g., comparison, filtering, aggregation, etc.) to be performed in the vector space without the need to calculate inverse trigonometric functions.

[0120] Based on this implementation, arctan calculation can be completely avoided, and vector representation naturally supports linear operations such as weighted averaging and convolution filtering. The following description further illustrates the scheme of this disclosure using the normalized gradient to represent the gradient direction as an example.

[0121] In some embodiments, clustering edge points based on the gradient direction and projection distance of the edge points to determine the geometric parameters of the target rectangle may include: normalizing the gradient corresponding to the edge points to obtain a normalized gradient; clustering the edge points based on the normalized gradient and projection distance to obtain the edge groups corresponding to the four sides of the target rectangle; and determining the geometric parameters of the target rectangle in a local coordinate system based on the edge groups, wherein the local coordinate system is established based on the target plane.

[0122] The server can use the method described above for each edge point. The corresponding gradient is normalized to obtain the normalized gradient. The direction of the gradient can be represented by the normalized gradient.

[0123] Furthermore, the server can construct edge points. With reference point radial vector between and using radial vector With normalized gradient The absolute value of the inner product is used to determine the edge points. With reference point Projected distance between , It can satisfy:

[0124] Wherein, the radial vector can be an edge point. To the reference point The vector, i.e. Alternatively, the radial vector can also be a reference point. To the edge point The vector, i.e. .

[0125] Based on edge points normalized gradient Distance from projection The server can cluster edge points. By clustering edge points with the same normalized gradient and projection distance or those that are close within a preset range into a group, the edge groups corresponding to the four sides of the target rectangle in the local coordinate system can be obtained.

[0126] Furthermore, the server can determine the geometric parameters of the target rectangle in the local coordinate system of the target plane based on the clustered edge groups. For example, it can determine the four vertices (or corners) of the rectangle based on the edge groups, and then determine the geometric parameters such as the center, length, width, and rotation angle of the target and the plane.

[0127] Based on this implementation, edge points are clustered using normalized gradients and projected distances to obtain the edge groups corresponding to the four sides of the target rectangle. This transforms the rectangle edge group separation problem into a clustering problem based on normalized gradients and projected distances, without requiring continuous or complete boundaries, and exhibits strong robustness to noise, holes, or burrs in the point cloud. Furthermore, by determining the geometric parameters of the target rectangle based on edge groups in a two-dimensional local coordinate system, the rectangle fitting problem in three-dimensional space is reduced to a two-dimensional plane solution. This reduces computational complexity and facilitates the introduction of geometric constraints such as "opposite sides are parallel, adjacent sides are orthogonal," ensuring the consistency and accuracy of the output geometric parameters.

[0128] In some embodiments, clustering edge points based on the normalized gradient and projection distance of edge points may include: constructing a multidimensional feature vector corresponding to the edge points, wherein the multidimensional feature vector is composed of the coordinate components of the normalized gradient of the edge points and the projection distance; and clustering the edge points based on the multidimensional feature vector.

[0129] Servers can be based on edge points normalized gradient Distance from projection Construction and edge points Corresponding multidimensional feature vector ,in, , For normalized gradient The coordinate components in the x and y directions of the local coordinate system, respectively. The server can determine the coordinate components based on the edge points. Corresponding multidimensional feature vector The three corresponding components , , Simultaneously identical or simultaneously approximating within their respective preset ranges will result in similar multidimensional feature vectors. edge point They are clustered into the same sidegroup.

[0130] Based on this implementation, by constructing a multi-dimensional feature vector composed of normalized gradient coordinate components and projected distance, and clustering edge points based on this feature vector, the gradient direction discrimination capability and the projected distance discrimination capability can be integrated into a unified feature representation. This allows for the separation of the four sides of a rectangle and the separation of edge points at different levels of nested rectangles in a single clustering operation. Furthermore, the normalized gradient coordinate components naturally form a separation along the x-axis and y-axis in the feature space within the local coordinate system; the projected distance feature further distinguishes edges with the same direction but different distances from the reference point (such as opposite sides of the same rectangle, or inner and outer sides of nested rectangles with the same direction) in the feature space. The combined use of both in the same feature vector ensures that the clustering results stably correspond to the actual side groups of the rectangle regardless of its rotation angle and size, demonstrating strong adaptability to changes in the rectangle's posture.

[0131] In some embodiments, clustering edge points based on the normalized gradient and projection distance of edge points may include: performing a uniformization process on the normalized gradient of the edge points based on the radial vector between the edge points and the reference points to obtain a processed normalized gradient, such that the processed normalized gradient and the radial vector satisfy a preset directional relationship, wherein the uniformization process includes maintaining the normalized gradient direction unchanged or reversing the normalized gradient; and clustering the edge points based on the processed normalized gradient and projection distance of the edge points.

[0132] The server can perform uniformization on the normalized gradient. The specific operations of uniformization can include: maintaining the direction of the normalized gradient unchanged, i.e., maintaining... Unchanged, will As the processed normalized gradient ; or reverse the normalized gradient, that is As the processed normalized gradient .

[0133] The server can perform uniform processing on the normalized gradients of all edge points, so that the processed normalized gradients and the radial vector satisfy a preset directional relationship.

[0134] Optionally, the preset directional relationship can be the angle between the processed normalized gradient and the radial vector. Within the range, at this time it is satisfied .

[0135] Optionally, the preset directional relationship can be the angle between the processed normalized gradient and the radial vector. Within the range, at this time it is satisfied .

[0136] After the homogenization process, the processed normalized gradients of the edge points can all face towards the reference point side, or all face away from the reference point side.

[0137] For example, when the radial vector is the vector from an edge point to the reference point (i.e., ), if the processed normalized gradient satisfies , then the processed normalized gradient faces towards the reference point side; if the processed normalized gradient satisfies , then the processed normalized gradient faces away from the reference point side.

[0138] Another example is when the radial vector is the vector from the reference point to the edge point (i.e., ). If the processed normalized gradient satisfies , then the processed normalized gradient faces away from the reference point side; if the processed normalized gradient satisfies , then the processed normalized gradient faces towards the reference point side.

[0139] Refer to Figure 6 , Figure 6 which is a schematic diagram of the homogenization process of the normalized gradient provided by an embodiment of this application.

[0140] As Figure 6 shown, for a "hui"-shaped target rectangle with nested rectangles, according to the gradient definition, the gradient direction of the edge points on its outer boundary can point inward, while the gradient direction of the edge points on its inner boundary can point outward.

[0141] Figure 6 shows an example where the reference point is inside the target rectangle, and after the homogenization process of the normalized gradient, the normalized gradients (i.e., the gradient directions) all face towards the reference point side.

[0142] The specific operation of the server can be, for example, for each , assuming its radial vector . If the normalized gradient and the radial vector satisfy , then reverse the normalized gradient, and use as the processed normalized gradient . At this time, all the processed normalized gradients Pointing to reference point One side, that is, the inside of the rectangle.

[0143] After the standardization process, the server can cluster edge points based on the processed normalized gradient and projected distance of the edge points.

[0144] Alternatively, the server can first use DBSCAN clustering: setting the neighborhood radius. (For example, 0.3) and minimum number of points (For example, 10), in a three-dimensional feature space ( Clustering was performed in the data to obtain several clusters. Each cluster corresponds to a set of points with consistent gradient directions and similar projection distances.

[0145] Alternatively, the server can employ mean shift or spectral clustering to partition similar point groups in the feature space.

[0146] Then, the server can calculate the principal direction of all edge points. and its orthogonal direction And for each cluster, the processed normalized gradient is calculated. average gradient and edge point center .like Then it is classified as either P edge or R edge (parallel to). Otherwise, it belongs to either Q or S edge (parallel to). ).

[0147] Next, the server can access the edge points of each cluster, for example, through the center. The orientation relative to the global center of the target rectangle is further used to determine whether it is P or R, etc.

[0148] Ultimately, the server can obtain four edge group sets. , , , Each set can contain multiple subgroups (such as the inner and outer sides of nested rectangles).

[0149] Based on this implementation, by selectively inverting the normalized gradient for consistency processing, it can be ensured that the normalized gradients of all edge points on the same edge have the same sign convention (or directional semantics) in the direction. This allows the gradient direction component to accurately reflect the orientation information of the edge when used as a clustering feature, avoiding the incorrect classification of points on the same edge into different clusters in the feature space due to inconsistent gradient signs, or the incorrect classification of points belonging to different edges into the same cluster.

[0150] In some embodiments, clustering edge points based on their normalized gradients and projected distances may include: standardizing the normalized gradients of edge points based on the radial vector between the edge point and a reference point to obtain processed normalized gradients, such that the processed normalized gradients and the radial vectors satisfy a preset directional relationship; constructing multidimensional feature vectors corresponding to the edge points, wherein the multidimensional feature vectors are composed of the coordinate components of the processed normalized gradients of the edge points and the projected distance; and clustering the edge points based on the multidimensional feature vectors. Each step of this embodiment has been described in detail in the corresponding or similar steps of other embodiments above, and will not be repeated here.

[0151] In some embodiments, determining the geometric parameters of a target rectangle in a local coordinate system based on edge groups may include: establishing a system of equations associated with the four sides of the target rectangle in the local coordinate system according to the geometric constraints of the rectangle based on the edge groups; solving the system of equations to obtain the four corner points of the target rectangle; and determining the geometric parameters of the target rectangle based on the four corner points.

[0152] The server obtains the four edge group sets. , , , Then, based on these edge sets, and according to the rectangular geometric constraints (e.g., opposite sides are parallel, adjacent sides are orthogonal, etc.), a system of equations related to the four sides of the target rectangle can be established in the local coordinate system. By solving the system of equations, the server can obtain the four corner points (i.e., vertices) of the target rectangle, and thus determine the geometric parameters of the target rectangle, such as the geometric center, length, width, and in-plane rotation angle of the target rectangle.

[0153] Specifically, the server can generate sampling index combinations from , , , Choose one subgroup from each of the four groups (4 groups in total), and then randomly select from each subgroup. A set of 2 points (e.g., 2 by default) forms a minimum sampling set. This mechanism can iterate through all possible combinations, ensuring that all potential rectangle candidates are covered.

[0154] Optionally, , , , Only the top three subgroups with the most points in the four edge groups can be retained to avoid combination explosion during the rectangular sampling combination process.

[0155] Optionally, if a rectangle's length / width range is predefined, the server can perform size prior filtering, thereby quickly eliminating combinations that do not meet the size requirements and improving efficiency.

[0156] Optionally, if it is not possible to obtain non-empty values ​​at the same time... , , , If there are four sides, the server can determine that there is no valid rectangle in the current input, return a failure flag, and optionally output debugging information.

[0157] For each set of sampling points, the server can establish a system of equations in the target plane that are associated with the four sides of the target rectangle. The system of equations can be expressed as:

[0158] in, and satisfy: .

[0159] Based on the established system of equations, the server can construct a global least squares problem: and about Solve the problem.

[0160] Optionally, the server can use Singular Value Decomposition (SVD) to solve the problem and process the last two digits of the solution vector. Normalize and then substitute back to obtain .

[0161] Alternatively, the server can directly solve the constrained quadratic optimization problem using generalized eigenvalue decomposition or the Lagrange multiplier method.

[0162] After solving for the equations of the four sides of the rectangle, the server can calculate the four corner points of the rectangle from the intersection of the lines, and then obtain the two-dimensional geometric center, width, length, and rotation angle in the plane. Equal geometric parameters.

[0163] Optionally, the server can also perform RANSAC iterative evaluation: by using the rectangular parameter model obtained from the current solution equations, the distance from each original 3D point to the spatial plane rectangle is calculated. This distance can be defined as the combination of the distance from the 3D point to the plane containing the target rectangle and the distance from the point to the region enclosed by the four sides of the rectangle.

[0164] By statistically analyzing distances less than a threshold The number of inliers is used to calculate the fitness (inlier ratio). The server can record the rectangular parametric model with the highest fitness as the best model; if the fitness is the same, the rectangular parametric model with the smallest root mean square error of inliers can be recorded as the best model.

[0165] Optionally, the server can also dynamically update the required number of iterations according to RANSAC theory. :

[0166] Where p is the confidence probability (e.g., 0.99999), w is the current proportion of inliers, and m is the minimum number of sampling points.

[0167] Based on this implementation, by establishing a system of equations in the local coordinate system related to the four sides of the target rectangle based on edge groups and rectangular geometric constraints, the geometric constraints of "parallel opposite sides and orthogonal adjacent sides" of the rectangle are explicitly encoded into the solution process. This fundamentally ensures that the solved rectangle satisfies geometric consistency, overcoming problems such as vertex misalignment, non-straight edges, and non-square corners caused by noise in existing polygon-based fitting schemes. Furthermore, by solving the system of equations to obtain the four corner points of the target rectangle, and determining the geometric parameters including the geometric center, length, width, and in-plane rotation angle based on the four corner points, a complete closed-form solution of the rectangle's geometric parameters can be achieved. This results in high computational efficiency and is suitable for real-time industrial deployment scenarios.

[0168] It is understood that gradient directions obtained by any other suitable method (e.g., angle doubling, complex representation, gradient structure tensor, etc. mentioned above) can be used in conjunction with the projection distance to cluster edge points to determine the geometric parameters of the target rectangle, similar to the gradient directions determined by the normalized gradient as described above. This application will not elaborate further on the comparison.

[0169] S206. Determine the attitude parameters of the target rectangle based on the geometric parameters and the normal vector of the target plane.

[0170] The normal vector of the target plane can be directly obtained from the aforementioned plane fitting steps, and the normal vector can be used to characterize the spatial orientation of the target plane.

[0171] The server can combine geometric parameters with the normal vector of the target plane, and determine the pose parameters of the target rectangle in 3D space based on the inverse projection transformation mapping. The pose parameters can include the spatial position of the target rectangle, the plane normal vector, and the rotation angle of the rectangle in the plane, thus forming a spatial pose description of the target rectangle and outputting the final model parameters. ,in, To map the center of the target rectangle in the local coordinate system back to the center of the rectangle in three-dimensional space through an inverse projection transformation, Let be the normal vector of the target plane for plane fitting. The width of the target rectangle. The length of the target rectangle. It represents the rotation angle.

[0172] Optionally, the server may also output parameters such as interior point index and fitting score, which are not limited in this application.

[0173] Through this step, the server can map the geometric parameters obtained in the local coordinate system back to three-dimensional space, thereby achieving a complete attitude estimation of the target rectangle.

[0174] In some embodiments, the method may further include: obtaining a subset of three-dimensional point cloud from the three-dimensional point cloud based on the pose parameters of the target rectangle and a preset distance threshold; and performing pose estimation on the target rectangle based on the subset of three-dimensional point cloud to update the pose parameters of the target rectangle.

[0175] After obtaining the parameters of the target rectangle through the steps described above, the server can also perform fine-tuning optimization to obtain more accurate pose parameters.

[0176] Specifically, the server can perform fine-grained optimization steps: 1. Interior point re-filtering: Calculate the distance from all original 3D points to the attitude parameter model using the attitude parameter model, and retain those with a distance less than [specified value]. Points (e.g., 0.01m) are used as the final interior point set. .

[0177] 2. Fine plane fitting: for Perform plane fitting again, using a more stringent plane threshold. (e.g., 0.003m) to obtain a more accurate plane normal vector. and points in the plane.

[0178] 3. Reprojection and Edge Repartitioning: Points in the fine-grained plane are reprojected onto the local coordinate system with adaptive resolution to generate two-dimensional floating-point coordinates. Using the obtained rectangular edge directions, each two-dimensional point is assigned to the nearest edge (P, Q, R, S), forming a new edge group. Optionally, the server can also use RANSAC line fitting to perform line fitting on each cluster separately to obtain the edge groups corresponding to P, Q, R, and S.

[0179] 4. Constrained Least Squares Again: Perform constrained least squares fitting on the new edge group to obtain the optimized two-dimensional rectangle parameters.

[0180] 5. Update the 9-parameter model, where: center of rectangle The optimized 2D center is mapped back to 3D space through inverse projection transformation; normal vector : Take the normal vector of the fine plane fitting; Width w and length l: obtained from the distance between the corner points of the three-dimensional rectangle; Rotation angle : Calculated from the direction vector of the two-dimensional rectangle.

[0181] 6. Output: Final model parameters , as well as interior point index and fit score.

[0182] Based on this implementation, a subset of 3D point cloud data is obtained from the 3D point cloud by using the pose parameters of the target rectangle and a more refined preset distance threshold. Inliers that highly match the current best model are selected, eliminating outliers and background interference, significantly improving the quality of the point cloud data used for subsequent pose estimation. By performing pose estimation on the target rectangle based on the 3D point cloud subset to update the pose parameters, a refined optimization of the initial estimation results is achieved. This further improves the accuracy and stability of the final output pose parameters compared to estimations based on all inliers. Furthermore, this refinement step can form a cascaded architecture for the entire pose estimation process, effectively improving the final pose estimation quality without significantly increasing computational cost. This is particularly suitable for industrial measurement and robot guidance scenarios with high pose accuracy requirements.

[0183] It is understood that, based on the method of this application, by performing planar fitting on the three-dimensional point cloud corresponding to the target rectangle, extracting edge points based on the set of interior points of the target plane, constructing joint features of direction and distance, and then solving the geometric parameters of the rectangle, the rectangular target in the spatial point cloud is transformed into a two-dimensional local geometric problem for processing. This can reduce the solution complexity of directly performing high-dimensional fitting in three-dimensional space, effectively distinguish each side of the target rectangle, and achieve complete and accurate estimation of attitude parameters.

[0184] Figure 7 This is a flowchart illustrating another spatial planar rectangular pose estimation method provided in an embodiment of this application. The execution entity of this method can be the aforementioned terminal device or server. Alternatively, Figure 7 The method shown can be executed by a chip in a terminal device or server; this application does not limit this. For ease of description, Figure 7 The following explanation uses the server as the execution subject of the method.

[0185] 701. Adaptive resolution calculation and coarse plane fitting.

[0186] After the server obtains the 3D point cloud of the target rectangle to be estimated, it can search for the nearest neighbor distance of each point in the 3D point cloud with respect to its neighboring points, calculate the average nearest neighbor distance for all points, use this average nearest neighbor distance as the subsequent projection resolution, and coarsely fit the target plane based on the RANSAC method.

[0187] 702. Floating-point projection and integer mask generation.

[0188] The server can project a 3D point cloud onto a coarsely fitted 2D target plane, determine and retain the floating-point coordinate system of the points projected onto the target plane based on the resolution of the adaptive calculation, generate an integer mask (i.e., integer coordinates) based on the floating-point coordinates, and create and maintain a mapping table between the floating-point coordinate system and the integer mask.

[0189] 703. Morphological Closure Operations and Gradient Calculation.

[0190] The server can first fill the holes or discontinuities between the integer masks corresponding to the target rectangle by performing morphological closing operations, and then calculate the gradient direction and magnitude of the processed integer masks with respect to the target rectangle to extract the edge point set. .

[0191] 704. Orientation-distance dual-feature clustering.

[0192] The server can perform a uniformization process on the gradient directions of the edge points to ensure that the gradient directions have the same directional semantics with respect to the reference point. For example, when the reference point is the geometric center of the edge points, the processed gradient directions of all edge points will point inwards towards the target rectangle. After the uniformization process, the server can calculate the projected distance between the reference point and the edge points along the gradient direction. Furthermore, based on the uniformized gradient directions and projected distances, the server can calculate the projected distances between the reference point and the edge points regarding the set of edge points. Construct a direction-distance multidimensional feature space, and cluster edge points in this feature space based on the DBSCAN method to classify edge points into P / Q / R / S edge groups.

[0193] 705. Determine whether edge group P / Q / R / S is non-empty.

[0194] The server can determine whether edge group P / Q / R / S is non-empty. If edge group P / Q / R / S is not empty, then execute 706; if edge group P / Q / R / S contains an empty set, then execute 709.

[0195] 706. Constrained least squares rectangle fitting.

[0196] The server can determine the equation set of the edge group P / Q / R / S through sampling combinations based on the RANSAC method, and solve the equation set by... The least squares problem with constraints is iterated to obtain the optimal rectangular model and determine the spatial orientation parameters of the target rectangle.

[0197] 707. Refined optimization.

[0198] The server can use the interior points of the rectangular model in the 3D point cloud to perform a rigorous plane fitting based on the iteratively obtained rectangular model, re-divide the edge groups, and perform constrained least squares again to refine the rectangular model.

[0199] 708. Output attitude parameters.

[0200] The server can output the pose parameters of the target rectangle based on the updated rectangular model. The parameters may include at least one of the following: rectangle center, normal vector, length, rotation angle, and fitting score.

[0201] 709. Return failed.

[0202] The server can return a failure message indicating that no valid rectangle exists.

[0203] Figure 8 This is a schematic diagram of the structure of an apparatus according to an embodiment of this application. Figure 8 The device 800 shown may be a terminal device or a server, or a device within a terminal device or server, or a device that can be used in conjunction with a terminal device or server.

[0204] Figure 8 The apparatus 800 shown may include an acquisition unit 801 and a processing unit 802.

[0205] The acquisition unit 801 can perform the corresponding acquisition function to acquire the data to be processed.

[0206] The processing unit 802 is used to process data, which may be data acquired by the acquisition unit 801. The processing unit 802 is also used to perform the data processing function in the foregoing method embodiments to achieve the corresponding processing function.

[0207] Optionally, the device 800 further includes a storage module, which can be used to store instructions and / or data; the processing unit 802 can read the instructions and / or data in the storage module so that the device 800 can implement the aforementioned method embodiments.

[0208] In one possible design, the device 800 may correspond to the terminal device or server in the above-described key dimension acquisition method embodiments, or to a component (such as a circuit, chip, or chip system) configured in the terminal device or server. The device 800 may be used to perform the steps or processes executed by the terminal device or server in any of the spatial planar rectangular pose estimation method embodiments in the first aspect described above.

[0209] For example, the acquisition unit 801 is used to acquire the 3D point cloud corresponding to the target rectangle.

[0210] The processing unit 802 is used to perform plane fitting on the 3D point cloud to extract the target plane where the target rectangle is located and the set of interior points in the 3D point cloud that belong to the target plane.

[0211] The acquisition unit 801 is also used to acquire the gradient of each interior point in the set of interior points in order to determine the edge points of the target rectangle.

[0212] The processing unit 802 is further configured to, for each edge point, determine the projection distance between the edge point and the reference point in the target plane in its gradient direction; cluster the edge points based on the gradient direction and projection distance corresponding to the edge points to determine the geometric parameters of the target rectangle, wherein the geometric parameters include at least one of geometric center, length, width, and in-plane rotation angle; and determine the attitude parameters of the target rectangle based on the geometric parameters and the normal vector of the target plane.

[0213] Based on this implementation method, by performing planar fitting on the 3D point cloud corresponding to the target rectangle, extracting edge points based on the set of interior points of the target plane, constructing joint features of direction and distance, and then solving the geometric parameters of the rectangle, the rectangular target in the spatial point cloud is transformed into a two-dimensional local geometric problem for processing. This can reduce the solution complexity of directly performing high-dimensional fitting in 3D space, effectively distinguish the sides of the target rectangle, and achieve complete and accurate estimation of attitude parameters.

[0214] In some implementations, reference points include: the geometric center of all edge points; the local center of some edge points; and reference points determined based on historical poses.

[0215] In some implementations, edge points are clustered based on the gradient direction and projection distance corresponding to the edge points to determine the geometric parameters of the target rectangle. This includes: normalizing the gradient corresponding to the edge points to obtain normalized gradients; clustering the edge points based on the normalized gradients and projection distances to obtain the edge groups corresponding to the four sides of the target rectangle; and determining the geometric parameters of the target rectangle in a local coordinate system based on the edge groups.

[0216] In some implementations, edge points are clustered based on their normalized gradients and projected distances. This includes: normalizing the normalized gradients of edge points based on the radial vectors between the edge points and reference points to obtain processed normalized gradients, such that the processed normalized gradients and the radial vectors satisfy a preset directional relationship. The normalization process includes maintaining the direction of the normalized gradients unchanged or reversing the normalized gradients; and clustering the edge points based on their processed normalized gradients and projected distances.

[0217] In some implementations, edge points are clustered based on their normalized gradients and projected distances. This includes: constructing multidimensional feature vectors corresponding to the edge points, where the multidimensional feature vectors are composed of the coordinate components of the normalized gradients of the edge points and the projected distances; and clustering the edge points based on the multidimensional feature vectors.

[0218] In some implementations, the geometric parameters of the target rectangle are determined in the local coordinate system based on the edge group, including: establishing a system of equations related to the four sides of the target rectangle in the local coordinate system according to the geometric constraints of the rectangle based on the edge group; solving the system of equations to obtain the four corner points of the target rectangle; and determining the geometric parameters of the target rectangle based on the four corner points.

[0219] In some implementations, obtaining the gradient of each interior point in the set of interior points includes: projecting all interior points in the set onto a local coordinate system to obtain the two-dimensional coordinates of each interior point, wherein the local coordinate system is established based on the target plane; and obtaining the gradient of each interior point based on the two-dimensional coordinates of each interior point.

[0220] In some implementations, the gradient of each interior point is obtained based on its two-dimensional coordinates, including: determining the projection resolution based on the statistical value of the minimum nearest neighbor distance of each point in the three-dimensional point cloud; determining the two-dimensional floating-point coordinates of each interior point based on its two-dimensional coordinates and the projection resolution; and determining the gradient of each interior point based on its two-dimensional floating-point coordinates.

[0221] In some implementations, the gradient of each interior point is determined based on its two-dimensional floating-point coordinates, including: generating two-dimensional integer coordinates of each interior point based on its two-dimensional floating-point coordinates; and determining the gradient of each interior point based on its two-dimensional integer coordinates.

[0222] In some implementations, the method further includes: obtaining a subset of the 3D point cloud from the 3D point cloud based on the pose parameters of the target rectangle and a preset distance threshold; and performing pose estimation on the target rectangle based on the subset of the 3D point cloud to update the pose parameters of the target rectangle.

[0223] The above are merely examples; for detailed steps or procedures, please refer to the descriptions in the foregoing embodiments.

[0224] Figure 9 This is a schematic diagram of another device provided in an embodiment of this application. Device 900 can be a terminal device or server as described in the above method embodiments, or it can be a chip, chip system, or processor that supports the terminal device or server in implementing the above methods. This device can be used to implement the methods described in the above embodiments of key dimension acquisition or edge placement error acquisition; for details, please refer to the descriptions in the above method embodiments.

[0225] like Figure 9As shown, the device 900 may include one or more processors 901. A processor 901, also called a processing unit or processing module, can implement certain control functions. The processor 901 can be a general-purpose processor or a dedicated processor, such as a central processing unit (CPU). The CPU can be used to control the device, execute software programs, and process data from the software programs.

[0226] In an alternative design, the processor 901 may also store instructions and / or data, which can be executed by the processor 901 to cause the device 900 to perform the methods described in the above method embodiments.

[0227] Optionally, the device 900 may include one or more memories 902, which may store a first instruction 903. The first instruction 903 may be executed on the processor 901, causing the device 900 to perform the method described in the above method embodiments. Optionally, the memory 902 may also store data. The processor 901 and the memory 902 may be provided separately or integrated together.

[0228] In another possible design, the processor 901 may optionally store a second instruction 904, which, when executed on the processor 901, causes the device 900 to perform the method described in the above method embodiments. The second instruction 904 may be embedded in the processor 901; in this case, the processor 901 may be implemented in hardware.

[0229] Optionally, the device 900 may further include a transceiver 905. The transceiver 905, which may be referred to as a transceiver unit or transceiver circuit, is used to implement transceiver functions. The transceiver 905 may include a receiver and a transmitter. The receiver, which may be referred to as a receiver circuit or receiving device, is used to implement a receiving function; the transmitter, which may be referred to as a transmitter circuit or transmitting device, is used to implement a transmitting function. The transceiver circuits, interfaces, interface circuits, or transceivers used to implement the receiving and transmitting functions may be separate or integrated. The aforementioned transceiver circuits, interfaces, interface circuits, or transceivers can be used for reading and writing code / data, or they can be used for transmitting or relaying signals.

[0230] In another possible design, the processor 901 may include a transceiver for implementing receive and transmit functions. For example, the transceiver may be a transceiver circuit, an interface, or an interface circuit. The transceiver circuit, interface, or interface circuit for implementing receive and transmit functions may be separate or integrated. The aforementioned transceiver circuit, interface, or interface circuit may be used for reading and writing code / data, or for transmitting or relaying signals.

[0231] It should be understood that, in one possible design, the steps in the method embodiments provided in this application can be implemented by integrated logic circuits in the processor's hardware or by instructions in software form. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are not provided here.

[0232] In one implementation, the device 900 may correspond to the server or terminal device in the above method embodiments, and may be used to execute the various steps and / or processes executed by the server or terminal device in the above method embodiments. The processor 901 may be used to execute instructions stored in the memory 902, and when the processor 901 executes the instructions stored in the memory, the processor 901 is used to execute the various steps and / or processes of the above method embodiments.

[0233] The apparatus described in the above embodiments may be a server or a terminal device, but the scope of the apparatus described in the embodiments of this application is not limited thereto, and the structure of the apparatus may vary. Figure 9 The device is subject to limitations. It can be a standalone device or part of a larger device. For example, the device could be: (1) Independent integrated circuit IC, or chip, or chip system or subsystem; (2) A collection of one or more ICs, optionally including a storage component for storing data and instructions; (3) ASIC; (4) Modules that can be embedded in other devices; (5) Others, etc.

[0234] For cases where the sensor device can be a chip or a chip system, please refer to [link / reference]. Figure 10 The diagram shows the structure of the chip. Figure 10 The chip 1000 shown includes a processor 1001 and an interface 1002. Optionally, it may also include a memory 1003. The number of processors 1001 can be one or more, and the number of interfaces 1002 can be multiple.

[0235] For the case where the chip is used to implement the spatial planar rectangular attitude estimation method of the projection exposure device in the embodiments of this application: Interface 1002 is used to receive or output signals; Processor 1001 is used to perform data processing operations on the sensor.

[0236] It is understood that some optional features in the embodiments of this application can be implemented independently in certain scenarios without relying on other features, such as the current solution on which they are based, to solve the corresponding technical problems and achieve the corresponding effects. Alternatively, they can be combined with other features as needed in certain scenarios. Accordingly, the sensor device given in the embodiments of this application can also implement these features or functions, which will not be elaborated here.

[0237] It should be understood that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method embodiments can be completed by the integrated logic circuitry in the processor's hardware or by instructions in software form. The processor described above can be a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a system-on-chip (SoC), a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0238] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0239] This application also provides a chip system including one or more processors for calling and executing instructions stored in memory, thereby performing the methods described in the embodiments of this application. The chip system may be composed of chips or may include chips and other discrete devices.

[0240] This application also provides a computer program product including instructions, the computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform the various steps or processes performed by the terminal device or server in any of the foregoing method embodiments.

[0241] This application also provides a computer-readable storage medium storing program code that, when run on a computer, causes the computer to perform the various steps or processes executed by the terminal device or server in any of the foregoing method embodiments.

[0242] The computer-readable storage medium may be the aforementioned volatile memory or non-volatile memory, or it may include both volatile memory and non-volatile memory.

[0243] In the embodiments of this application, the terms and English abbreviations are exemplary examples given for ease of description and should not be construed as limiting the application in any way. This application does not preclude the possibility of defining other terms that can achieve the same or similar functions in existing or future agreements.

[0244] The above embodiments can be implemented entirely or partially by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated.

[0245] A computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. A computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0246] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0247] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some operations can be performed in other orders or simultaneously. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0248] The descriptions of the various embodiments provided in this application can be referenced mutually. Each embodiment has its own emphasis, and parts not described in detail in a certain embodiment can be referred to the relevant descriptions of other embodiments. For the sake of convenience and brevity, for example, the functions and operations of the various devices and equipment provided in the embodiments of this application can be referred to the relevant descriptions of the method embodiments of this application. The method embodiments and the device embodiments can also be referenced, combined or cited from each other.

[0249] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method of spatial planar rectangular pose estimation, the method comprising: The method comprises: obtaining a three-dimensional point cloud corresponding to a target rectangle; plane fitting is performed on the three-dimensional point cloud to extract a target plane on which the target rectangle is located and a set of inliers in the three-dimensional point cloud that belong to the target plane; obtaining the gradient of each inlier in the set of inliers to determine the edge points of the target rectangle; for each edge point, determining the projection distance between the edge point and a reference point in the target plane in the gradient direction of the edge point; based on the gradient direction and the projection distance corresponding to the edge point, clustering the edge points to determine the geometric parameters of the target rectangle, wherein the geometric parameters include at least one of the geometric center, the length, the width, and the in-plane rotation angle; based on the geometric parameters and the normal vector of the target plane, determining the pose parameters of the target rectangle.

2. The method of claim 1, wherein, The clustering of the edge points based on the gradient direction and the projection distance corresponding to the edge points to determine the geometric parameters of the target rectangle comprises: normalizing the gradient corresponding to the edge point to obtain a normalized gradient; based on the normalized gradient and the projection distance of the edge point, clustering the edge points to obtain a side group corresponding to each side of the target rectangle; based on the side group, determining the geometric parameters of the target rectangle in a local coordinate system, wherein the local coordinate system is established based on the target plane.

3. The method of claim 2, wherein, The clustering of the edge points based on the normalized gradient and the projection distance of the edge point comprises: based on the radial vector between the edge point and the reference point, performing consistent processing on the normalized gradient of the edge point to obtain a processed normalized gradient, so that the processed normalized gradient and the radial vector satisfy a preset directional relationship, wherein the consistent processing includes maintaining the direction of the normalized gradient or reversing the normalized gradient; based on the processed normalized gradient and the projection distance of the edge point, clustering the edge points.

4. The method of claim 2, wherein, The clustering of the edge points based on the normalized gradient and the projection distance of the edge point comprises: constructing a multi-dimensional feature vector corresponding to the edge point, wherein the multi-dimensional feature vector is composed of the coordinate components of the normalized gradient and the projection distance of the edge point; based on the multi-dimensional feature vector, clustering the edge points.

5. The method of claim 2, wherein, The determination of the geometric parameters of the target rectangle in the local coordinate system based on the side group comprises: based on the side group, establishing an equation set associated with the four sides of the target rectangle in the local coordinate system according to the geometric constraints of the rectangle; solving the equation set to obtain four corner points of the target rectangle; based on the four corner points, determining the geometric parameters of the target rectangle.

6. The method of claim 1, wherein, The obtaining of the gradient of each inlier in the set of inliers comprises: projecting all inliers in the set to a local coordinate system to obtain two-dimensional coordinates of each inlier, wherein the local coordinate system is established based on the target plane; based on the two-dimensional coordinates of each inlier, obtaining the gradient of each inlier.

7. The method of claim 6, wherein, The obtaining of the gradient of each inlier based on the two-dimensional coordinates of each inlier comprises: determining a projection resolution based on a statistical value of the minimum neighborhood distance of each point in the three-dimensional point cloud; Based on the two-dimensional coordinates of each interior point and the projection resolution, determine the two-dimensional floating-point coordinates of each interior point; The gradient of each interior point is determined based on its two-dimensional floating-point coordinates.

8. The method of claim 7, wherein, The determination of the gradient of each interior point based on the two-dimensional floating-point coordinates of each interior point includes: Based on the two-dimensional floating-point coordinates, generate two-dimensional integer coordinates for each interior point; The gradient of each interior point is determined based on the two-dimensional integer coordinates of each interior point.

9. The method of claim 1, wherein, The reference points include: The geometric center of all edge points; Local center of some edge points; Reference points determined based on historical posture.

10. The method of claim 1, wherein, The method further includes: Based on the pose parameters of the target rectangle and a preset distance threshold, a subset of the three-dimensional point cloud is obtained from the three-dimensional point cloud; The pose of the target rectangle is estimated based on the subset of the 3D point cloud to update the pose parameters of the target rectangle.

11. An apparatus, comprising: Includes units for performing the method as described in any one of claims 1-10.

12. An apparatus, comprising: It includes a processor and a memory, the processor and the memory being coupled, the processor being used to implement the method as described in any one of claims 1-10.

13. A chip, characterized by The device includes a processor and an interface, the processor and the interface being coupled; the interface is used to receive or output signals, and the processor is used to execute code instructions to cause the method of any one of claims 1-10 to be performed.

14. A storage medium, characterized by The storage medium stores computer-executable instructions, which, when invoked by a computer, cause the computer to perform the method described in any one of claims 1-10.

Images