Three-dimensional measurement method of strongly light-reflecting objects and related apparatus

Abstract

This disclosure relates to a three-dimensional measurement method and related equipment for highly reflective objects. The method includes: generating a three-dimensional point cloud of the object based on the point cloud corresponding to each of multiple surfaces; wherein, for each surface, the point cloud corresponding to that surface is acquired through a first iteration; for each first iteration, the following operations are performed: calculating a translation vector based on the point cloud corresponding to the beginning of the current first iteration; acquiring the point cloud after the robot translation; fusing the point cloud corresponding to the beginning of the current first iteration with the point cloud acquired after the robot translation; if there is no saturation boundary within the fused point cloud, the fused point cloud is used as the point cloud corresponding to that surface; otherwise, the fused point cloud is used as the point cloud corresponding to the beginning of the next first iteration. By controlling the robot to perform active movement, it can autonomously avoid saturation regions on the object, ensuring the acquisition of accurate and complete point clouds, thus improving the accuracy of three-dimensional measurement.

Description

Technical Field

[0001] This disclosure relates to the field of structured light vision 3D measurement technology, and more specifically, to 3D measurement methods and related equipment for highly reflective objects. Background Technology

[0002] Autonomous intelligent robot operation can be applied in many fields, such as machining and grinding, navigation and positioning, reverse engineering, clamping and assembly, defect detection, etc. Acquiring complete and accurate 3D information of the target object is the core of achieving autonomous intelligent robot operation. Currently, 3D measurement of objects is mainly achieved by actively projecting coded patterns onto the object under test using structured light vision technology. However, the robustness and completeness of the point cloud obtained by the robot largely depend on the physical characteristics of the object's surface. For objects with strong reflectivity and no texture, such as machined metal parts, smooth plastics, or painted surfaces, the beam projected by the structured light system is prone to pixel saturation in the camera that acquires fringe modulation information after reflection from such surfaces. This makes it impossible to calculate the height information of the corresponding area of ​​the object, resulting in the loss of a large area of ​​point cloud, and consequently, difficulty in accurately obtaining the 3D information of such objects.

[0003] In related technologies, to achieve 3D reconstruction of highly reflective, textureless objects, researchers have proposed a series of high dynamic range structured light techniques, mainly including: changing camera exposure parameters, changing projection brightness, and using additional hardware. However, changing camera exposure parameters is difficult to achieve pixel-level brightness control, and selecting the optimal exposure parameters is challenging. Changing projection brightness requires projecting a large number of stripe sequences with different brightness levels to obtain an adaptive projection pattern, resulting in low measurement efficiency. Although adaptive local brightness adjustment methods can reduce the number of projections, reliably mapping saturated areas in the camera image onto the projection plane remains difficult. More importantly, in areas with particularly high reflectivity, even with minimal projection intensity to suppress reflection in highly reflective areas, the pixels corresponding to the centers of these bright areas in the camera-acquired image will still become saturated, leading to the loss of 3D information. Using additional hardware typically replaces the traditional camera in the structured light system with a polarization camera or light field camera, which reduces image resolution and increases system complexity. Therefore, there is an urgent need for a method to perform 3D measurements on objects with highly reflective and textureless characteristics to obtain more accurate 3D information. Summary of the Invention

[0004] This disclosure provides a three-dimensional measurement method and related equipment for highly reflective objects, so as to at least solve the problem in the above-mentioned related technologies that it is difficult to accurately obtain the three-dimensional information of objects with strong reflectivity and no texture.

[0005] According to a first aspect of the present disclosure, a three-dimensional measurement method for a highly reflective object is provided, applied to a structured light system. The structured light system includes a robot, a projector, and a camera. The projector and the camera are mounted at the end of the robot's robotic arm. The method includes: acquiring point clouds corresponding to each of a plurality of faces of the object; generating a three-dimensional point cloud of the object based on the point clouds corresponding to each of the plurality of faces; wherein, for each face, the point cloud corresponding to that face is acquired through a first iteration; for each first iteration, the following operations are performed until the first iteration terminates: acquiring the point cloud corresponding to the start of the current first iteration, wherein, during the initial execution of the first iteration, the point cloud is measured based on the stripes projected by the projector. After the image is projected onto the object, the point cloud acquired by the camera based on the stripe image projected onto the object is used to determine the point cloud corresponding to the start of the first iteration; based on the point cloud corresponding to the start of the first iteration, a translation vector is calculated; in response to the robot translating according to the translation vector, the point cloud after the robot's translation is acquired; the point cloud corresponding to the start of the first iteration is fused with the point cloud acquired after the robot's translation to obtain a fused point cloud; in response to the absence of a saturation boundary in the fused point cloud, the fused point cloud is used as the point cloud corresponding to that surface; in response to the presence of a saturation boundary in the fused point cloud, the fused point cloud is used as the point cloud corresponding to the start of the next first iteration.

[0006] Optionally, for a preset surface of the object, when the first iteration is performed for the first time, the point cloud corresponding to the start of the first iteration is the point cloud collected from a preset initial viewpoint.

[0007] Optionally, the three-dimensional measurement method further includes: for each of the surfaces of the object other than the preset surface, performing density clustering on the boundary point clouds of saturated regions in multiple point clouds to obtain multiple point cloud clusters, wherein the multiple point clouds include: all point clouds acquired and collected in each first iteration for each surface before the current surface; calculating the viewpoint corresponding to each point cloud cluster in the multiple point cloud clusters to obtain multiple viewpoints; sorting the multiple viewpoints according to the density of the multiple point cloud clusters from largest to smallest to obtain a sorting result; for each of the other surfaces, calculating the point cloud corresponding to the start of the first iteration through a second iteration; wherein, for each second iteration, performing the following operations until the second iteration terminates: confirming... Determine the viewpoint at the start of the current second iteration and collect point clouds from that viewpoint, wherein the viewpoint at the start of the first second iteration is the first viewpoint in the sorting result; calculate the first distance between the center of the point cloud collected in the current second iteration and the center of a preset fused point cloud, wherein the preset fused point cloud refers to the fused point cloud obtained in the last first iteration for a face preceding the current face; in response to the first distance being greater than or equal to a preset threshold, use the point cloud collected in the current second iteration as the point cloud corresponding to the start of the first iteration when the first iteration is first performed for the current face; in response to the first distance being less than the preset threshold, select the next viewpoint adjacent to the viewpoint at the start of the current second iteration from the sorting result as the viewpoint at the start of the next second iteration.

[0008] Optionally, calculating the viewpoint corresponding to each of the plurality of point cloud clusters includes: constructing a reference coordinate system based on each point cloud cluster, wherein the center of the point cloud cluster is the origin of the reference coordinate system, the normal vector of the point cloud cluster is the z-axis of the reference coordinate system, the principal direction obtained by principal component analysis of the point cloud cluster is the y-axis of the reference coordinate system, and the secondary direction obtained is the x-axis of the reference coordinate system; and calculating the viewpoint corresponding to the point cloud cluster based on the reference coordinate system.

[0009] Optionally, the three-dimensional measurement method further includes: for each first iteration, performing the following operations: determining a second distance between the center of the point cloud corresponding to the start of the current first iteration and the plane of the projector; calculating a rotation angle based on the second distance; performing a cross product between the translation vector and the principal axis vector of the projector to obtain a rotation axis; calculating a rotation matrix based on the rotation axis and the rotation angle; and acquiring the point cloud after the robot rotates according to the rotation matrix; wherein fusing the point cloud corresponding to the start of the current first iteration with the point cloud acquired after the robot's translation includes: fusing the point cloud corresponding to the start of the current first iteration, the point cloud acquired after the robot's translation, and the point cloud acquired after the robot's rotation to obtain the fused point cloud.

[0010] Optionally, calculating the translation vector based on the point cloud at the start of the first iteration includes: extracting multiple saturated region boundary point clouds from the point cloud at the start of the first iteration; calculating the three-dimensional oriented bounding box of each saturated region boundary point cloud and determining the vector of the shortest side among the multiple sides contained in the three-dimensional oriented bounding box; and calculating the translation vector based on the vectors of the multiple shortest sides corresponding to the multiple saturated region boundary point clouds.

[0011] According to a second aspect of the present disclosure, a three-dimensional measurement device for a highly reflective object is provided, applied to a structured light system. The structured light system includes a robot, a projector, and a camera. The projector and the camera are mounted at the end of the robot's robotic arm. The device includes: a single-face point cloud acquisition module configured to acquire a point cloud corresponding to each face of a plurality of faces of an object; and a three-dimensional point cloud generation module configured to generate a three-dimensional point cloud of the object based on the point cloud corresponding to each face of the plurality of faces. For each face, the single-face point cloud acquisition module is configured to acquire the point cloud corresponding to that face through a first iteration. For each first iteration, the following operations are performed until the first iteration terminates: acquiring the point cloud corresponding to the start of the current first iteration, wherein, in the initial... During the first iteration, the point cloud corresponding to the start of the first iteration is determined based on the point cloud acquired by the camera after the projector projects the stripe image onto the object. A translation vector is calculated based on the point cloud at the start of the first iteration. In response to the robot translating according to the translation vector, the point cloud after the robot's translation is acquired. The point cloud at the start of the first iteration is fused with the point cloud acquired after the robot's translation to obtain a fused point cloud. If there is no saturation boundary within the fused point cloud, the fused point cloud is used as the point cloud corresponding to that surface. If there is a saturation boundary within the fused point cloud, the fused point cloud is used as the point cloud corresponding to the start of the next first iteration.

[0012] Optionally, for a preset surface of the object, when the first iteration is performed for the first time, the point cloud corresponding to the start of the first iteration is the point cloud collected from a preset initial viewpoint.

[0013] Optionally, the three-dimensional measurement device for the highly reflective object further includes: a density clustering module, configured to perform density clustering on the boundary point clouds of saturated regions in multiple point clouds for each of the other faces of the object besides the preset face, to obtain multiple point cloud clusters, wherein the multiple point clouds include: all point clouds acquired and collected in each first iteration for each face before the current face; a viewpoint calculation module, configured to calculate the viewpoint corresponding to each point cloud cluster in the multiple point cloud clusters, to obtain multiple viewpoints; a viewpoint sorting module, configured to sort the multiple viewpoints in descending order of density of the multiple point cloud clusters, to obtain a sorting result; and a second iteration module, configured to calculate the point cloud corresponding to the start of the first iteration for each of the other faces through a second iteration; wherein, for each second iteration... The second iteration module is configured to perform the following operations until the second iteration terminates: determine the viewpoint at the start of the current second iteration, and acquire a point cloud at the viewpoint, wherein the viewpoint at the start of the initial second iteration is the first viewpoint in the sorting results; calculate a first distance between the center of the point cloud acquired in the current second iteration and the center of a preset fused point cloud, wherein the preset fused point cloud refers to the fused point cloud obtained in the last first iteration for a face preceding the current face; in response to the first distance being greater than or equal to a preset threshold, use the point cloud acquired in the current second iteration as the point cloud at the start of the first iteration when the first iteration is first performed for the current face; in response to the first distance being less than the preset threshold, select the next viewpoint adjacent to the viewpoint at the start of the current second iteration from the sorting results as the viewpoint at the start of the next second iteration.

[0014] Optionally, the viewpoint calculation module is configured to: construct a reference coordinate system based on each point cloud cluster, wherein the center of the point cloud cluster is the origin of the reference coordinate system, the normal vector of the point cloud cluster is the z-axis of the reference coordinate system, the principal direction obtained by principal component analysis of the point cloud cluster is the y-axis of the reference coordinate system, and the secondary direction obtained is the x-axis of the reference coordinate system; and calculate the viewpoint corresponding to the point cloud cluster based on the reference coordinate system.

[0015] Optionally, the three-dimensional measurement device for the highly reflective object further includes: a distance determination module configured to determine a second distance between the center of the point cloud corresponding to the start of the first iteration and the plane of the projector; a rotation angle calculation module configured to calculate a rotation angle based on the second distance; a cross product module configured to perform a cross product between the translation vector and the principal axis vector of the projector to obtain a rotation axis; a rotation matrix calculation module configured to calculate a rotation matrix based on the rotation axis and the rotation angle; a rotating point cloud acquisition module configured to acquire the point cloud after the robot rotates according to the rotation matrix in response to the robot rotating; and a single-sided point cloud acquisition module configured to fuse the point cloud corresponding to the start of the first iteration, the point cloud acquired after the robot's translation, and the point cloud acquired after the robot's rotation to obtain the fused point cloud.

[0016] Optionally, the single-sided point cloud acquisition module is configured to: extract multiple saturated region boundary point clouds from the point cloud corresponding to the start of the first iteration; calculate the three-dimensional oriented bounding box of each saturated region boundary point cloud in the multiple saturated region boundary point clouds, and determine the vector of the shortest side among the multiple sides contained in the three-dimensional oriented bounding box; and calculate the translation vector based on the vectors of the multiple shortest sides corresponding one-to-one with the multiple saturated region boundary point clouds.

[0017] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a three-dimensional measurement method for highly reflective objects according to the present disclosure.

[0018] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform a three-dimensional measurement method for highly reflective objects according to the present disclosure.

[0019] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a three-dimensional measurement method for highly reflective objects according to the present disclosure.

[0020] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0021] In this disclosure, for each of the multiple faces of an object, the point cloud of that face can be calculated by iteratively controlling the robot to move multiple times. That is, this disclosure allows the robot to actively move and autonomously avoid saturated regions, i.e., highly reflective regions, on the object. This ensures the acquisition of accurate and complete point clouds, thereby guaranteeing that the measured 3D information of the object matches the actual situation of the object, improving the accuracy of 3D measurements for highly reflective, textureless objects.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0024] Figure 1 This is a schematic diagram illustrating a structured light system in an actual measurement scenario according to an exemplary embodiment of the present disclosure;

[0025] Figure 2 This is a schematic diagram illustrating a plurality of modules included in a structured light system according to an exemplary embodiment of the present disclosure;

[0026] Figure 3 This is a flowchart illustrating a three-dimensional measurement method for a highly reflective object according to an exemplary embodiment of the present disclosure;

[0027] Figure 4 This is a schematic diagram illustrating optical modeling of a measurement scene according to an exemplary embodiment of the present disclosure;

[0028] Figure 5 This is a schematic diagram illustrating an adaptive structured light process according to an exemplary embodiment of the present disclosure;

[0029] Figure 6 This is a schematic diagram illustrating the active movement of a robot to avoid areas of strong reflection according to an exemplary embodiment of the present disclosure;

[0030] Figure 7 This is a schematic diagram illustrating robot viewpoint generation according to exemplary embodiments of the present disclosure;

[0031] Figure 8 This is a schematic diagram illustrating a test object according to an exemplary embodiment of the present disclosure;

[0032] Figure 9 This is a schematic diagram illustrating measurement results obtained from performing three-dimensional measurements on a highly reflective, textureless object according to an exemplary embodiment of the present disclosure;

[0033] Figure 10 This is a block diagram illustrating a three-dimensional measuring device for a highly reflective object according to an exemplary embodiment of the present disclosure;

[0034] Figure 11 This is a block diagram illustrating an electronic device according to exemplary embodiments of the present disclosure. Detailed Implementation

[0035] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0036] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0037] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. As another example, "performing at least one of step one and step two" indicates the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0038] Figure 1 This is a schematic diagram illustrating a structured light system in an actual measurement scenario according to an exemplary embodiment of the present disclosure.

[0039] Reference Figure 1A structured light system primarily comprises a structured light sensor and a robot. The structured light sensor typically includes a projector and an industrial camera. The projector projects coded fringe images onto the target object, while the industrial camera captures the modulated coded fringe images—specifically, the coded fringe images reflected from the target object's surface. The robot carries the structured light sensor and actively moves to avoid highly reflective areas on the target object's surface. This allows for the construction of a complete 3D point cloud of the target object based on point clouds collected from multiple viewpoints during the active movement.

[0040] Figure 2 This is a schematic diagram illustrating a plurality of modules included in a structured light system according to an exemplary embodiment of the present disclosure.

[0041] Reference Figure 2 The structured light system mainly includes a "measuring scene optical modeling module", an "adaptive structured light module", a "robot strong reflection avoidance module" and a "robot viewpoint planning module". By executing the above modules in sequence, accurate 3D reconstruction of the surface of strongly reflective, textureless objects can be achieved.

[0042] The "Measurement Scene Optical Modeling Module" can use physically based rendering technology to model the structured light measurement scene, and then obtain the influence of the surface reflectivity of the object to be measured on the camera saturation from the energy perspective, thus providing a theoretical basis for the adaptive structured light module.

[0043] The "adaptive structured light module" mainly includes three parts: "adaptive projection brightness generation," "saturation region matching," and "3D point cloud generation." Its main function is to acquire local point clouds of highly reflective object surfaces. Specifically, the structured light system can project coded stripes and uniform brightness sequences onto the object surface, and can trigger a camera to synchronously capture and store images reflected from the object surface.

[0044] "Adaptive projection brightness generation" can be implemented based on a convolutional neural network. The network input can be an image captured under a uniform brightness projection sequence, and the output can be the optimal projection intensity map of the camera plane. "Saturation region matching" can be based on the correspondence between edge pixels of saturated regions in the image and their corresponding projector positions. This calculates the homography relationship between the saturated regions of the camera image and the corresponding regions on the projector's projection plane, thus obtaining the optimal projection intensity map of the projector plane. "3D point cloud generation" can be achieved by decoding the adaptively coded image of the object's surface height modulated by the camera, thereby obtaining the pixel matching relationship between the camera image and the projector image. Furthermore, by combining the intrinsic and extrinsic parameters of the camera and projector, a 3D point cloud of the measured object's surface can be calculated.

[0045] The "robot high-reflectivity avoidance module" mainly comprises three parts: "translational motion," "rotational motion," and "compensation motion." Its primary function is to acquire a complete local point cloud of the highly reflective object's surface. Specifically:

[0046] Translational motion can be achieved by moving the structured light sensor along the object's surface with minimal displacement to precisely avoid the original highly reflective areas, thus ensuring that missing point cloud regions are adjacent and do not overlap. Furthermore, since translational motion alters the camera's field of view, rotational motion can maximize the consistency of the 3D measurement area. However, rotational motion changes the projector's orientation, which alters the beam's incident angle on the measurement surface, leading to changes in the reflection positions on the object's surface and the positions of saturated regions in the image, thus reducing the accuracy of avoiding highly reflective areas. Therefore, additional translational motion can be used to compensate for the displacement of reflective regions in the image.

[0047] The "Robot Viewpoint Planning Module" mainly includes "Viewpoint Generation" and "Viewpoint Adjustment". Specifically: Viewpoint generation can predict the robot's next observation posture based on existing point cloud information, thereby enabling effective exploration of unknown areas of the object under test; Viewpoint adjustment can allow the selection of a suboptimal viewpoint to ensure the measurement process continues when the generated viewpoints are in almost the same position.

[0048] This disclosure provides an improved solution from hardware to algorithms to achieve 3D reconstruction of highly reflective, textureless object surfaces. The solution integrates theoretical optical modeling, adaptive structured light algorithms, robot-assisted active avoidance of strong reflections, and point cloud-based viewpoint planning, achieving accurate, robust, and complete 3D reconstruction of large-sized, highly reflective, textureless objects, demonstrating strong practicality and engineering value.

[0049] Figure 3This is a flowchart illustrating a three-dimensional measurement method for a highly reflective object according to an exemplary embodiment of the present disclosure, applied to a structured light system. As previously described, the structured light system may include a robot, a projector, and a camera, and the projector and camera may be mounted at the end of a robot arm.

[0050] Reference Figure 3 In step 301, point clouds corresponding to each of the multiple faces of the object can be obtained. For example, assuming the object to be measured has 6 faces, point clouds corresponding to each of the 6 faces of the object can be obtained.

[0051] In step 302, a three-dimensional point cloud of an object can be generated based on the point cloud corresponding to each of the multiple faces. That is, the surface contour of the object to be measured can be reconstructed in three dimensions based on the point cloud corresponding to each of the multiple faces contained in the object to be measured.

[0052] Specifically, for each face, the point cloud corresponding to that face can be obtained through the first iteration.

[0053] Furthermore, for each first iteration, the following operations can be performed until the first iteration terminates:

[0054] First, the point cloud corresponding to the start of the first iteration can be obtained. Specifically, when the first iteration is executed for the first time, the point cloud corresponding to the start of the first iteration can be determined based on the point cloud collected by the camera after the stripe image is projected onto the object by the projector.

[0055] Then, a translation vector can be calculated based on the point cloud at the start of this first iteration. Next, in response to the robot translating according to the calculated translation vector, a point cloud can be acquired after the robot's translation. Then, the point cloud at the start of this first iteration can be fused with the point cloud acquired after the robot's translation to obtain a fused point cloud.

[0056] If there is no saturation boundary within the fused point cloud, the fused point cloud can be used as the point cloud corresponding to the surface; if there is a saturation boundary within the fused point cloud, the fused point cloud can be used as the point cloud corresponding to the start of the next first iteration.

[0057] It should be noted that the "saturation boundary" is the same as the "strong reflection region boundary," which can be considered a "hole" in the fused point cloud. If there is no saturation boundary within the fused point cloud, it means there is no strong reflection region within it, i.e., no "hole" within it. In this case, it indicates that the complete point cloud information for the corresponding surface has been obtained. Therefore, this fused point cloud can be directly used as the point cloud corresponding to that surface. If there is a saturation boundary within the fused point cloud, it means there is still a strong reflection region within it, i.e., there is still a "hole" within it. In this case, it indicates that the complete point cloud information for the corresponding surface has not yet been obtained. Therefore, this fused point cloud can be used as the point cloud corresponding to the start of the next first iteration after the current first iteration to continue the iteration process until the iteration terminates.

[0058] Figure 4 This is a schematic diagram illustrating optical modeling of a measurement scene according to an exemplary embodiment of the present disclosure. (Refer to...) Figure 4 Discrete modeling of the light energy received by each image pixel is possible. It's important to note that the intensity value of each image pixel is proportional to the light energy reflected back to that pixel in the camera. Furthermore, by considering factors such as ambient light, mutual reflection from object surfaces, camera noise, camera gain, and exposure time, a complete imaging model can be obtained. Further, the brightness of the projected pattern can be uniformly varied while acquiring images from the camera. Then, based on the imaging model, the reflectance of each image pixel region can be estimated using least-squares fitting.

[0059] It should be noted that each time the projector projects a striped image onto an object, it can perform two projections. The first projection is mainly used to obtain the reflectivity of various locations on the object. This reflectivity can be used to guide the adjustment of the projection brightness when the projector projects the striped image a second time. For example, areas with higher reflectivity can have their brightness reduced during the second projection, while areas with lower reflectivity can have their brightness increased, and so on.

[0060] Additionally, refer to Figure 4 In (b) of the diagram, in a light projection-receiving system, light propagation follows the principles of geometric optics. The position p of the object's surface can be modeled using the theory of microfacets, which represents it as a set of many small, ideal reflecting planes with random orientations at a microscopic scale. Furthermore, in... Figure 4 In (c), L ref The intensity of the reflected light, L dif For diffuse reflection light intensity, L absor The light intensity absorbed by the workpiece surface, L spe For the intensity of mirror reflection, This represents the normal vector at position p. In a structured light system, the reference... Figure 4In (c), the microsurface is defined within a spatial region corresponding to a single pixel. Discrete modeling can be performed on the light energy received by each pixel, where (x... c ,y c The value at point () is proportional to the light energy reflected back to the camera:

[0061]

[0062] Where position p is located at The incident radiation intensity in the direction is exist The reflection intensity in the direction is denoted as K g A constant representing the mapping relationship between light energy and pixel value, where α is reflectivity.

[0063] (x c ,y c ) represents the image coordinates, I(x) c ,y c ) is the point (x c ,y c The pixel intensity value is 0 to 255, where 0 represents complete black and 255 represents complete white.

[0064] Additionally, consider the ambient light component. and Mutually reflected light L from the surface of the object inter Camera noise I noise Factors such as camera gain k and exposure time t can be used to derive a complete camera imaging model:

[0065]

[0066] Among them, L in include and L inter (p), where For position p at The intensity of incident light radiation from the directional projector For position p at Ambient light incident radiation intensity, L inter (p) represents the incident radiation intensity of light reflected from the workpiece surface at position p.

[0067] Then, we can have:

[0068] I(x c ,y c )=αx1+βx2+I noise

[0069] in:

[0070] Furthermore, by collecting n sets of data for each pixel (I i ,x i1 ,x i2 By systematically changing the intensity of the projector, α and β can be estimated using the least squares method. Specifically, Q can be defined as:

[0071]

[0072] To optimize Q, we can set the partial derivatives with respect to α and β to zero, resulting in:

[0073]

[0074] Furthermore, we can define X = [x 11 x 12 ,x 21 x 22 ,…,x n1 x n2 B = [α,β] T I = [I1, I2, ..., I n ] T Therefore, the above formula can be reformulated as: X T XB = X T I.

[0075] If X T If X is invertible, then the estimates of B and I can be obtained using the following formulas:

[0076]

[0077] exist Figure 4 In (d) of the middle, Let dA represent the normal vector at the infinitesimal element p, and let w represent the area of ​​the infinitesimal element in the calculus. Ψ Indicates the direction of the incident ray.

[0078] The key advantage of this analysis lies in its ability to simulate the reflective properties of a target. Specifically, by defining the surface reflectivity α and optically modeling the structured light measurement scenario, the surface properties of an object can be effectively described. It should be noted that the surface reflectivity parameter α is a fundamental component controlling the projection behavior of the structured light system in highly reflective regions, and it can also provide a basis for robot motion planning to mitigate strong reflections.

[0079] Figure 5 This is a schematic diagram illustrating an adaptive structured light process according to an exemplary embodiment of the present disclosure. (Refer to...) Figure 5 (a) and Figure 5 In (b), the actively controllable parameters that have a significant impact on pixel values ​​are k, t, and L. projWe can fix k and t to constants k and t respectively. fix and t fix It can also adjust the projection brightness to achieve the ideal unsaturated pixel value I. exp In this way, the optimal projection intensity can be determined.

[0080]

[0081] Once I is set exp This allows calculation based on each pixel. However, when the pixel size is large, performing matrix inversion for each pixel leads to excessive computation. Therefore, a method based on convolutional neural networks can be introduced, which can take images captured under a uniform brightness projection sequence as input and quickly generate the optimal projection intensity map. Specifically, referring to... Figure 5 In (c), the network architecture can consist of two branches: the first branch (branch 1), the main branch, and the second branch (branch 2). The main branch (branch 1) can employ a multi-scale high-resolution feature extraction backbone paired with a self-focus module, processing multi-channel input images and identifying saturation regions. Then, these pixel locations can be classified into multiple classes to assign appropriate brightness categories. The second branch (branch 2) can extract features from the input image and predict specific brightness values ​​for each pixel class. Finally, the outputs of the two branches can be integrated to generate the optimal projection intensity map.

[0082] It should be noted that, It is defined within the camera's imaging plane. To effectively reduce reflections from the target surface, it is necessary to... Convert to This transformation can be achieved by calculating the homography matrix for each saturation region to ensure an accurate mapping of optimal projection intensity. Specifically, to achieve this, the projector can be modeled as an inverse camera, and it can be assumed that both the camera and the projector follow a pinhole imaging model.

[0083] s c (x c ,y c ,1) T =K c [R c |t c ](X w ,Y w Z w ) T

[0084] sp (x p ,y p ,1) T =K p [R p |t p ](X w ,Y w Z w ) T .

[0085] In this model, s c ,s p K is the scaling factor. c ,K p For the intrinsic parameters of the camera and projector, [R] c |t c ] and [R p |t p [x] represents the external parameters of the camera and projector. If the projected vertical fringes are known, i.e., x... p The direction code value is known, i.e., x. p Given that, for a point (x) in the known camera plane... c ,y c The decoded code value and x can be obtained through decoding. p Correspondingly, the goal is to simultaneously find the y-coordinate of this pixel in the projector plane. p This forms a point-to-point relationship. We can assume:

[0086]

[0087] Furthermore, (X) can be w ,Y w Z w As an intermediate value in the calculation process, it can then be solved to obtain:

[0088]

[0089] Currently only s remains p y p The unknowns are not known, but we can directly construct two equations to solve for these two unknowns separately:

[0090]

[0091] At this point, the edge pixels (x) of the saturated region in the image can be established. c ,y c ) and its corresponding projector position (x) p ,y p The correspondence between these correspondences allows us to construct the homography relationship of the saturation region from the camera's imaging plane to the projector's projection plane:

[0092] (x p ,y p ,1) T =H·(x c ,y c ,1) T

[0093] The matrix H can be solved by identifying four non-collinear pairs of points along the boundary of the saturated region. Finally, by... From (x) c ,y c Mapping to (x) p ,y p This will generate the corresponding local dark stripe pattern. The specific calculation process is as follows: Figure 5 (d) in the middle. Next, refer to Figure 5 In option (e), projection can be performed again according to the optimized brightness mode.

[0094] Furthermore, to enhance robustness, an encoding scheme integrating grayscale codes and line-shift codes can be employed. This strategy minimizes the impact of reflections during initial encoding mode design and decoding. Additionally, subpixel edge extraction can be utilized using stripes, which can further improve the accuracy of the point cloud.

[0095] It should be noted that the above framework is tailored for measurement scenarios of highly reflective targets. It comprehensively considers factors such as encoding mode selection, decoding, triangulation algorithm, efficient calculation of optimal projection intensity, and accurate mapping of saturated regions. Its main goal is to mitigate the adverse effects of high reflectivity on measurement accuracy at each algorithm stage.

[0096] Figure 6 This is a schematic diagram illustrating the active movement of a robot to avoid areas of strong reflection according to an exemplary embodiment of the present disclosure.

[0097] Reference Figure 6 As mentioned earlier, the robot's strong reflection avoidance module mainly includes translational motion, rotational motion, and compensating motion. The projector's beam is on X... a A strong reflection is produced at the location of the captured image pixel. The vicinity causes saturation. Then, the movement of the sensor causes the corresponding reflective area on the object to shift from X... a Move to X b Since moving the camera alters its field of view, rotation is necessary to maintain a relatively constant field of view. The saturation region after rotation is X. c At this time, at the pixel position of the captured image Saturation occurs in the vicinity. Finally, compensating motion can be performed to capture the pixel positions of the image. The surrounding area causes saturation, specifically:

[0098] Reference Figure 6 In (b), the translational motion can be achieved by translating the structured light sensor along the object surface with minimal displacement to precisely avoid the original highly reflective areas, thereby ensuring that the missing point cloud regions are adjacent and do not overlap. Furthermore, referring to... Figure 6 In (c), because translational motion alters the camera's field of view, rotational motion can also maximize the consistency of the 3D measurement area. Additionally, refer to... Figure 6 In (d), the rotational motion changes the projector's orientation, which alters the incident angle of the light beam on the measurement surface. This, in turn, changes the position of the reflection on the object's surface and the position of the saturated region in the image, resulting in a decrease in accuracy in avoiding highly reflective areas. Therefore, additional translational motion can be used to compensate for the displacement of reflective areas in the image.

[0099] According to an exemplary embodiment of this disclosure, multiple saturated region boundary point clouds can be extracted from the point cloud corresponding to the start of the first iteration. Then, a three-dimensional oriented bounding box for each of the multiple saturated region boundary point clouds can be calculated, and the vector of the shortest side among the multiple edges contained in the three-dimensional oriented bounding box can be determined. Furthermore, the surface area and center of the three-dimensional oriented bounding box can also be determined. Next, a translation vector can be calculated based on the vectors of the multiple shortest sides corresponding one-to-one with the multiple saturated region boundary point clouds. Specifically, the weighted value of all shortest side vectors can be used as the direction vector of the translation movement, and the projection value of the largest shortest side vector in the direction of translation movement can be used as the translation amount.

[0100] In this way, the robot's strong reflection avoidance module, through translational motion, can desaturate previously saturated surface areas in the image, allowing for the recovery of missing point clouds through additional measurements. Specifically, to minimize the movement distance, the point cloud acquired from the current viewpoint can be analyzed. For example, firstly, the boundaries of all saturated regions can be extracted from the striped image, and the point cloud corresponding to each saturated region boundary can be calculated. Then, for each A 3D oriented bounding box (OBB) can be calculated, and then the surface area of ​​each OBB can be obtained. center and the shortest edge vector Next, we can calculate the weighted unit vector by weighting all the shortest edge vectors. The direction of sensor movement is used as the reference point. Finally, the projection value of the largest shortest side vector onto the direction of translational motion can be used. As a quantity of translational motion:

[0101]

[0102] According to an exemplary embodiment of this disclosure, for each first iteration process, the following operations may also be performed:

[0103] Determine the second distance between the center of the point cloud at the start of the first iteration and the projector plane. Then, calculate the rotation angle based on this second distance. Next, perform a cross product between the translation vector and the projector's principal axis vector to obtain the rotation axis. Then, calculate the rotation matrix based on the rotation axis and rotation angle. For example, the Rodrigues rotation formula can be used to construct the rotation matrix. Next, in response to the robot rotating according to the rotation matrix, collect the point cloud after the robot's rotation. Then, fuse the point cloud at the start of the first iteration, the point cloud collected after the robot's translation, and the point cloud collected after the robot's rotation to obtain a fused point cloud.

[0104] It should be noted that the robot's strong reflection avoidance translational motion will change the camera's field of view and the projector's illumination range at the current viewpoint. Therefore, it is necessary to maintain the consistency of the 3D measurement area as much as possible through the robot's rotational motion. Specifically, let's assume that the rotation centers of the robot's end effector before and after the translational motion are denoted as O and O', respectively, and the distance from O to the projector is R, with the point cloud center C... ptc The distance to the projection plane before motion is D. Once the transformation projection from the projector to the camera is calibrated... and hand-eye matrix This allows us to directly determine the spatial geometric relationship between the robot and the sensors, as well as the relative relationship between their coordinate systems. Specifically, the robot's rotation angle θ can be calculated by establishing a reference plane. rot Furthermore, the normal to the reference plane can be determined by the cross product of the projector's principal axis vector and the translational motion vector, specifically expressed as: Furthermore, based on geometric relationships, we can conclude that:

[0105]

[0106] Where, α rot Projector principal axis vector with o′o p The included angle, o' is the rotation center of the robot's end effector, o p This is the optical center of the projector.

[0107] Given a robot's known pose, the only unknown in the above equation is θ. rot This can be determined through numerical calculation. Furthermore, to obtain the final homogeneous matrix of the rotational motion, the rotation axis and its direction can be defined as: Then, the rotation matrix R can be constructed by applying Rodrigues' rotation formula. rot :

[0108]

[0109] Furthermore, the robot's strong reflective avoidance rotational motion alters the projector's orientation, which changes the incident angle of the light beam on the measurement surface, thus shifting the reflection position on the surface from X... b Move to X c ,like Figure 6 As shown in (c) above. Simultaneously, the pose of the camera's imaging plane changes, causing saturation regions in the image to shift, which reduces the accuracy of avoiding highly reflective areas. To address this issue, the robot can also be guided to fine-tune the sensor's observation posture to compensate for the displacement of reflective regions in the image.

[0110] The compensation action can be performed in two steps. First, after calculating the displacement of the reflecting area on the surface, additional translational motion can be applied to reposition the reflecting area to its predetermined location. The amount of compensation motion can be the amount of displacement of the center of the saturation region caused by the translational and rotational motions. The amount of shift in the center of the saturation region caused by translational motion The difference between them. Then, the distance between the sensor and the object can be adjusted to maintain a standard distance. Specifically, after translation, rotation, and compensated motion, the sensor can move along... Repositioning to maintain point cloud center C ptc and projection plane Ω proj The distance between them is equal to D, and the adjustment of the distance can be denoted as...

[0111] According to an exemplary embodiment of this disclosure, for a preset surface of an object, when performing the first iteration for the first time, the point cloud corresponding to the start of the first iteration can be a point cloud acquired from a preset initial viewpoint. This preset surface can be the first surface of the object measured during the 3D measurement process.

[0112] According to an exemplary embodiment of this disclosure, for each of the faces of an object other than a preset face, density clustering can be performed on the boundary point clouds of saturated regions in multiple point clouds to obtain multiple point cloud clusters. These multiple point clouds can include all point clouds acquired and collected during each first iteration for each face preceding the current face. For example, suppose three faces of an object have already been measured, and a fourth face needs to be measured. In this case, density clustering can be performed on all point clouds acquired and collected during each first iteration for each of the three already measured faces. For instance, suppose the first face requires three first iterations to obtain its complete point cloud, the second face requires five first iterations to obtain its complete point cloud, and the third face requires four first iterations to obtain its complete point cloud. Then, density clustering can be performed on the point clouds acquired and collected in all first iterations (3+5+4=12 first iterations) corresponding to these three faces.

[0113] Then, the viewpoint corresponding to each point cloud cluster can be calculated, resulting in multiple viewpoints. For example, assuming a total of 10 point cloud clusters are obtained, 10 viewpoints can be calculated accordingly. Next, the multiple viewpoints can be sorted according to the density of the multiple point cloud clusters from largest to smallest, obtaining a sorting result. For example, the 10 viewpoints can be sorted according to the density of the 10 point cloud clusters from largest to smallest, obtaining a sorting result.

[0114] It should be noted that, for each of the other faces, the point cloud corresponding to the start of the first iteration can be calculated using the second iteration. Furthermore, for each second iteration, the following operations can be performed until the second iteration terminates:

[0115] First, the viewpoint at the start of this second iteration can be determined, and a point cloud can be acquired from that viewpoint. The viewpoint at the start of the first second iteration can be the first viewpoint in the aforementioned sorting results. For example, as mentioned earlier, assuming a total of 10 viewpoints are obtained, the viewpoint at the start of the first second iteration for the 4th face can be the first viewpoint among the aforementioned 10 sorted viewpoints.

[0116] Then, the first distance between the center of the point cloud acquired in this second iteration and the center of the preset fused point cloud can be calculated. The preset fused point cloud can refer to the fused point cloud obtained in the last first iteration for the face preceding the current face. For example, as mentioned above, the current face can be the fourth face, and the preset fused point cloud can refer to the fused point cloud obtained in the last first iteration for the third face.

[0117] Next, in response to the aforementioned first distance being greater than or equal to a preset threshold, the point cloud collected in this second iteration can be used as the point cloud corresponding to the start of the first iteration when the first iteration is first performed on the current face. For example, as mentioned above, the current face can be the fourth face, and the point cloud collected in this second iteration can be used as the point cloud corresponding to the start of the first iteration when the first iteration is first performed on the fourth face.

[0118] Optionally, in response to the aforementioned first distance being less than a preset threshold, the next viewpoint adjacent to the viewpoint at the start of the current second iteration can be selected from the sorting results as the viewpoint at the start of the next second iteration. For example, as mentioned earlier, assuming a total of 10 viewpoints are obtained, and the viewpoint at the start of the current second iteration is the first of these 10 viewpoints, then if the aforementioned first distance is less than the preset threshold, the second viewpoint adjacent to the first viewpoint at the start of the current second iteration can be selected from the sorted 10 viewpoints as the viewpoint at the start of the next second iteration, thereby allowing the measurement process to continue. Furthermore, the aforementioned first distance can be, but is not limited to, Euclidean distance.

[0119] It should be noted that the above example illustrates the implementation process of the second iteration using an object that has a total of 6 faces, with the current face being the 4th face. In reality, the number of faces an object can be arbitrary, for example, 3 faces, 8 faces, or more. Furthermore, the second iteration process for any face other than the 1st face of the object is similar to the second iteration process for the 4th face described above, and will not be repeated here. The aforementioned implementation method is merely an illustrative example.

[0120] As mentioned earlier, robot viewpoint generation primarily depends on the representation of point cloud density. Specifically, firstly, density clustering can be performed on the point cloud at the boundaries of saturated regions, and the points with the highest number of points can be clustered (denoted as...). This serves as the first reference cluster for viewpoint generation. A reference coordinate system can then be constructed based on this first reference cluster. Specifically, a coordinate system can be defined... center As the origin of the coordinate system, it can be defined The normal vector is used as Coordinate axes. Additionally, principal component analysis (PCA) can be used to extract... The primary and secondary directions can be identified, and the extracted primary and secondary directions can be assigned values ​​respectively: and Thus, a reference coordinate system can be obtained. Representation in the robot's reference coordinate system.

[0121] Figure 7 This is a schematic diagram illustrating robot viewpoint generation according to an exemplary embodiment of the present disclosure. (Refer to...) Figure 7 The projection plane of a projector has its major axis. Let be the principal axis of the projector coordinate system. Let be the minor axis of the projector's projection plane. Furthermore, to maximize coverage of the boundary area, Need to be aligned with the main direction Alignment allows the projector to build the next viewpoint:

[0122]

[0123] One advantage of this viewpoint generation method is that it ensures sufficient overlap between point clouds captured from consecutive viewpoints, which facilitates the registration and fusion of multiple point cloud frames in subsequent processes.

[0124] Furthermore, in robot viewpoint generation, when the target surface of an object has discontinuous regions or occlusions, the incremental new point cloud generated by measuring these regions is often insufficient. Additionally, the boundary regions of the merged point clouds from previous and current viewpoints may exhibit very small updates, potentially leading to the generation of subsequent viewpoints at almost the same locations, thus delaying the exploration process. To mitigate this, it is essential to ensure that these situations are detected promptly and that the viewpoint is adjusted accordingly to guarantee continuous and comprehensive observation of the target surface. This can be achieved by calculating the center C of the point cloud collected from the current viewpoint. i The center C of the fused point cloud obtained in the last first iteration is the previous face. i-1 The Euclidean distance between the two points is used as the criterion for whether the viewpoint needs to be adjusted and updated. Specifically: if the calculated Euclidean distance is less than a threshold D... th This indicates that there are special surface conditions that require triggering a viewpoint adjustment strategy. For example, viewpoint adjustment could involve determining the second largest cluster point as the suboptimal viewpoint based on the aforementioned point cloud clustering results and using it as the adjusted viewpoint, thereby allowing the measurement process to continue.

[0125] According to an exemplary embodiment of this disclosure, a reference coordinate system can be constructed based on each point cloud cluster. The center of the point cloud cluster can be the origin of the reference coordinate system, and the normal vector of the point cloud cluster can be the z-axis of the reference coordinate system. The principal direction obtained by performing principal component analysis on the point cloud cluster can be the y-axis of the reference coordinate system, and the secondary direction obtained can be the x-axis of the reference coordinate system. Then, the viewpoint corresponding to the point cloud cluster can be calculated based on the constructed reference coordinate system.

[0126] Figure 8 This is a schematic diagram illustrating a test object according to an exemplary embodiment of the present disclosure. (Refer to...) Figure 8The objects being tested can be highly reflective machined metal parts, convex pyramids with smooth paint, or concave bowls. These objects all share the characteristics of high reflectivity and a lack of surface texture.

[0127] Figure 9 This is a schematic diagram illustrating measurement results obtained from performing three-dimensional measurements on a highly reflective, textureless object according to an exemplary embodiment of the present disclosure. (Refer to...) Figure 9 The structured light system disclosed herein exhibits superior 3D reconstruction performance on all types of surfaces, including highly reflective and textureless objects, and also achieves high point cloud reconstruction density.

[0128] To address the difficulty in accurately measuring 3D objects with strong reflection and no texture in related technologies, this disclosure provides a robot active 3D measurement method based on structured light vision. This method has the following main advantages:

[0129] (1) The robot can be controlled to move actively to avoid strong reflective areas on the object, thereby ensuring the acquisition of complete local point cloud. The accumulated local point cloud can then be used to plan the viewpoint for comprehensive three-dimensional measurement of the object.

[0130] (2) By setting up the robot viewpoint planning module, it can be ensured that it is applicable to the three-dimensional measurement of large-sized, highly reflective, and textureless objects, and has a wide range of applications.

[0131] (3) The structured light system provided in this disclosure has simple hardware and can consist of only a projector, a camera and a robot, without any additional hardware equipment, and has a low cost.

[0132] Figure 10 This is a block diagram illustrating a three-dimensional measuring device 1000 for a highly reflective object according to an exemplary embodiment of the present disclosure. The three-dimensional measuring device 1000 can be applied to a structured light system, which may include a robot, a projector, and a camera. Furthermore, the projector and camera may be mounted at the end effector of the robot's robotic arm.

[0133] Reference Figure 10 The three-dimensional measuring device 1000 may include a single-sided point cloud acquisition module 1001 and a three-dimensional point cloud generation module 1002.

[0134] The single-face point cloud acquisition module 1001 can acquire the point cloud corresponding to each face of an object. For example, assuming the object to be measured has 6 faces, the point cloud corresponding to each of the 6 faces of the object can be acquired.

[0135] The 3D point cloud generation module 1002 can generate a 3D point cloud of an object based on the point cloud corresponding to each of the multiple faces. That is, it can perform 3D reconstruction of the surface contour of the object to be measured based on the point cloud corresponding to each of the multiple faces contained in the object to be measured.

[0136] Specifically, for each face, the single-face point cloud acquisition module 1001 can acquire the point cloud corresponding to that face through a first iteration. Furthermore, for each first iteration, the single-face point cloud acquisition module 1001 can perform the following operations until the first iteration terminates:

[0137] First, the point cloud corresponding to the start of the first iteration can be obtained. Specifically, when the first iteration is executed for the first time, the point cloud corresponding to the start of the first iteration can be determined based on the point cloud collected by the camera after the stripe image is projected onto the object by the projector.

[0138] Then, a translation vector can be calculated based on the point cloud at the start of this first iteration. Next, in response to the robot translating according to the calculated translation vector, a point cloud can be acquired after the robot's translation. Then, the point cloud at the start of this first iteration can be fused with the point cloud acquired after the robot's translation to obtain a fused point cloud.

[0139] If there is no saturation boundary within the fused point cloud, the fused point cloud can be used as the point cloud corresponding to the surface; if there is a saturation boundary within the fused point cloud, the fused point cloud can be used as the point cloud corresponding to the start of the next first iteration.

[0140] According to an exemplary embodiment of this disclosure, the single-sided point cloud acquisition module 1001 can extract multiple saturated region boundary point clouds from the point cloud corresponding to the start of the current first iteration. Then, a three-dimensional oriented bounding box for each of the multiple saturated region boundary point clouds can be calculated, and the vector of the shortest side among the multiple sides contained in the three-dimensional oriented bounding box can be determined. Further, the surface area and center of the three-dimensional oriented bounding box can also be determined. Next, a translation vector can be calculated based on the vectors of the multiple shortest sides corresponding one-to-one with the multiple saturated region boundary point clouds. Specifically, the weighted value of all shortest side vectors can be used as the direction vector of the translation movement, and the projection value of the largest shortest side vector in the direction of translation movement can be used as the translation amount.

[0141] According to an exemplary embodiment of this disclosure, the three-dimensional measuring device 1000 may further include a distance determination module, a rotation angle calculation module, a cross product module, a rotation matrix calculation module, and a rotation point cloud acquisition module.

[0142] For each first iteration, the following operations can also be performed:

[0143] The distance determination module is configured to determine a second distance between the center of the point cloud at the start of the first iteration and the plane of the projector. Then, the rotation angle calculation module is configured to calculate the rotation angle based on the second distance. Next, the cross product module is configured to perform a cross product between the translation vector and the principal axis vector of the projector to obtain the rotation axis. Then, the rotation matrix calculation module is configured to calculate the rotation matrix based on the rotation axis and the rotation angle. For example, the rotation matrix can be constructed using the Rodrigues rotation formula. Next, in response to the robot rotating according to the rotation matrix, the rotation point cloud acquisition module can acquire the point cloud after the robot's rotation. Then, the single-sided point cloud acquisition module 1001 can fuse the point cloud at the start of the first iteration, the point cloud acquired after the robot's translation, and the point cloud acquired after the robot's rotation to obtain a fused point cloud.

[0144] According to an exemplary embodiment of this disclosure, for a preset surface of an object, when performing the first iteration for the first time, the point cloud corresponding to the start of the first iteration can be a point cloud acquired from a preset initial viewpoint. This preset surface can be the first surface of the object measured during the 3D measurement process.

[0145] According to an exemplary embodiment of this disclosure, the three-dimensional measuring device 1000 may further include a density clustering module, a viewpoint calculation module, a viewpoint sorting module, and a second iteration module.

[0146] For each face of an object other than the preset face, the density clustering module can perform density clustering on the boundary point clouds of saturated regions in multiple point clouds to obtain multiple point cloud clusters. The multiple point clouds can include: all point clouds acquired and collected in each first iteration for each face before the current face.

[0147] Then, the viewpoint calculation module can calculate the viewpoint corresponding to each point cloud cluster in multiple point cloud clusters, obtaining multiple viewpoints. Next, the viewpoint sorting module can sort the multiple viewpoints according to the density of the multiple point cloud clusters from largest to smallest, obtaining the sorting result.

[0148] It should be noted that, for each of the other faces, the second iteration module can calculate the point cloud corresponding to the start of the first iteration using the second iteration method. Furthermore, for each second iteration, the second iteration module can perform the following operations until the second iteration terminates:

[0149] First, the viewpoint at the start of this second iteration can be determined, and point clouds can be collected from that viewpoint. The viewpoint at the start of the first second iteration can be the first viewpoint in the aforementioned sorting results.

[0150] Then, the first distance between the center of the point cloud collected in this second iteration and the center of the preset fused point cloud can be calculated. The preset fused point cloud can refer to the fused point cloud obtained in the last first iteration for a face before the current face.

[0151] Next, in response to the aforementioned first distance being greater than or equal to a preset threshold, the point cloud collected in this second iteration can be used as the point cloud corresponding to the start of the first iteration when the first iteration is first executed for the current face.

[0152] Optionally, in response to the aforementioned first distance being less than a preset threshold, the next viewpoint adjacent to the viewpoint at the start of the current second iteration can be selected from the sorting results as the viewpoint at the start of the next second iteration. Furthermore, the aforementioned first distance can be, but is not limited to, Euclidean distance.

[0153] According to an exemplary embodiment of this disclosure, the aforementioned viewpoint calculation module can construct a reference coordinate system based on each point cloud cluster. The center of the point cloud cluster can be the origin of the reference coordinate system, and the normal vector of the point cloud cluster can be the z-axis of the reference coordinate system. The principal direction obtained by performing principal component analysis on the point cloud cluster can be the y-axis of the reference coordinate system, and the obtained secondary direction can be the x-axis of the reference coordinate system. Then, the viewpoint calculation module can calculate the viewpoint corresponding to the point cloud cluster based on the constructed reference coordinate system.

[0154] Figure 11 This is a block diagram illustrating an electronic device 1100 according to an exemplary embodiment of the present disclosure.

[0155] Reference Figure 11 The electronic device 1100 includes at least one memory 1101 and at least one processor 1102. The at least one memory 1101 stores instructions that, when executed by the at least one processor 1102, perform a three-dimensional measurement method for a highly reflective object according to an exemplary embodiment of the present disclosure.

[0156] As an example, electronic device 1100 may be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned instructions. Here, electronic device 1100 is not necessarily a single electronic device, but may be a collection of any devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 1100 may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0157] In electronic device 1100, processor 1102 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0158] The processor 1102 can execute instructions or code stored in the memory 1101, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transmission protocol.

[0159] The memory 1101 may be integrated with the processor 1102, for example, by arranging RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 1101 may include a separate device, such as an external disk drive, a storage array, or other storage device usable by any database system. The memory 1101 and the processor 1102 may be operatively coupled, or may communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 1102 to read files stored in the memory.

[0160] In addition, electronic device 1100 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of electronic device 1100 can be interconnected via a bus and / or network.

[0161] According to exemplary embodiments of this disclosure, a computer-readable storage medium may also be provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the aforementioned three-dimensional measurement method for highly reflective objects. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R, BD-R The computer program can be stored in a computer-readable storage medium such as a BD-RE, Blu-ray or optical disc storage device, hard disk drive (HDD), solid-state drive (SSD), card storage (such as a multimedia card, secure digital (SD) card, or ultra-fast digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, or any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0162] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, including a computer program that, when executed by a processor, implements the three-dimensional measurement method for highly reflective objects according to the present disclosure.

[0163] According to the 3D measurement method and related equipment for highly reflective objects disclosed herein, for each of the multiple faces of an object, the point cloud of that face can be calculated by iteratively controlling a robot to move multiple times. In other words, this disclosure allows the robot to actively move and autonomously avoid saturated areas, i.e., highly reflective areas, on the object. This ensures the acquisition of accurate and complete point clouds, thereby guaranteeing that the measured 3D information of the object matches the actual situation of the object, and improving the accuracy of 3D measurement for highly reflective, textureless objects.

[0164] According to exemplary embodiments of this disclosure, the robot's strong reflection avoidance translational motion changes the camera's field of view and the projector's illumination range at the current viewpoint. Therefore, it is necessary to maintain the consistency of the three-dimensional measurement area to the greatest extent possible through the robot's rotational motion.

[0165] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0166] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for three-dimensional measurement of a strongly light-reflecting object, applied to a structured light system, characterized in that, The structured light system comprises a robot, a projector and a camera, the projector and the camera are installed at the end of the mechanical arm of the robot, and the method comprises: Obtaining a point cloud corresponding to each of a plurality of faces of an object; Generating a three-dimensional point cloud of the object based on the point cloud corresponding to each of the plurality of faces; Wherein, for each face, the point cloud corresponding to the face is obtained by means of first iteration; For each first iteration process, the following operations are performed until the first iteration terminates: Obtaining the point cloud corresponding to the beginning of the first iteration, wherein, when the first iteration is performed for the first time, the point cloud corresponding to the beginning of the first iteration is determined based on the point cloud collected by the camera after the projector projects a stripe image onto the object and the camera captures the stripe image projected onto the object; Based on the point cloud corresponding to the beginning of the first iteration, a translation vector is calculated; In response to the robot translating according to the translation vector, a point cloud is collected after the robot translates; Fusing the point cloud corresponding to the beginning of the first iteration and the point cloud collected after the robot translates to obtain a fused point cloud; In response to the absence of a saturated boundary in the fused point cloud, the fused point cloud is taken as the point cloud corresponding to the face; In response to the presence of a saturated boundary in the fused point cloud, the fused point cloud is taken as the point cloud corresponding to the beginning of the next first iteration; Wherein, the three-dimensional measurement method further comprises: For each of the other faces of the object except for the preset face, the saturated area boundary point cloud in the plurality of point clouds is density clustered to obtain a plurality of point cloud clusters, wherein the plurality of point clouds include all point clouds obtained and collected in each first iteration process for each face before the current face; The view point corresponding to each point cloud cluster in the plurality of point cloud clusters is calculated to obtain a plurality of view points; The plurality of view points are sorted in descending order of the density of the plurality of point cloud clusters to obtain a sorting result; For each of the other faces, the point cloud corresponding to the beginning of the first iteration when the first iteration is performed for the first time is calculated by means of second iteration; Wherein, for each second iteration process, the following operations are performed until the second iteration terminates: Determining the view point at the beginning of the second iteration, and collecting a point cloud at the view point, wherein the view point at the beginning of the first iteration is the first view point in the sorting result; Calculating a first distance between the center of the point cloud collected in the second iteration and the center of a preset fused point cloud, wherein the preset fused point cloud refers to the fused point cloud obtained in the last first iteration for a face before the current face; In response to the first distance being greater than or equal to a preset threshold, the point cloud collected in the second iteration is taken as the point cloud corresponding to the beginning of the first iteration when the first iteration is performed for the first time for the current face; In response to the first distance being less than the preset threshold, the next view point adjacent to the view point at the beginning of the second iteration is selected from the sorting result as the view point at the beginning of the next second iteration.

2. The three-dimensional measurement method according to claim 1, wherein For the preset face, when the first iteration is performed for the first time, the point cloud corresponding to the start of the first iteration is a point cloud collected from a preset initial viewpoint.

3. The three-dimensional measurement method of claim 1, wherein, The calculating the viewpoint corresponding to each point cloud cluster in the plurality of point cloud clusters comprises: based on each point cloud cluster, a reference coordinate system is constructed, wherein the center of the point cloud cluster is the origin of the reference coordinate system, the normal vector of the point cloud cluster is the z-axis of the reference coordinate system, the principal direction obtained by performing principal component analysis on the point cloud cluster is the y-axis of the reference coordinate system, and the secondary direction obtained is the x-axis of the reference coordinate system; based on the reference coordinate system, the viewpoint corresponding to the point cloud cluster is calculated.

4. The three-dimensional measurement method of claim 1, wherein, The three-dimensional measurement method further comprises: For each first iteration process, the following operations are also performed: determine the second distance between the center of the point cloud corresponding to the start of the first iteration and the plane of the projector; based on the second distance, the rotation angle is calculated; the translation vector is cross-multiplied with the main axis vector of the projector to obtain a rotation axis; based on the rotation axis and the rotation angle, a rotation matrix is calculated; in response to the robot rotating according to the rotation matrix, the point cloud after the robot rotates is collected; wherein the fusion of the point cloud corresponding to the start of the first iteration and the point cloud collected after the robot translates comprises: fuse the point cloud corresponding to the start of the first iteration, the point cloud collected after the robot translates, and the point cloud collected after the robot rotates to obtain the fused point cloud.

5. The three-dimensional measurement method of claim 1, wherein, The calculation of the translation vector based on the point cloud corresponding to the start of the first iteration comprises: extract a plurality of saturated region boundary point clouds from the point cloud corresponding to the start of the first iteration; calculate a three-dimensional directional bounding box of each saturated region boundary point cloud in the plurality of saturated region boundary point clouds, and determine the vector of the shortest side of the three-dimensional directional bounding box; based on the plurality of shortest side vectors corresponding to the plurality of saturated region boundary point clouds, the translation vector is calculated.

6. A three-dimensional measuring device of a strongly light-reflecting object applied to a structured light system, characterized by, The structured light system comprises a robot, a projector and a camera, the projector and the camera are installed at the end of the mechanical arm of the robot, and the device comprises: a single-face point cloud acquisition module configured to acquire a point cloud corresponding to each face of a plurality of faces of an object; a three-dimensional point cloud generation module configured to generate a three-dimensional point cloud of the object based on the point cloud corresponding to each face of the plurality of faces; wherein for each face, the single-face point cloud acquisition module is configured to acquire the point cloud corresponding to the face by means of first iteration; for each first iteration process, the following operations are performed until the first iteration terminates: acquire the point cloud corresponding to the start of the first iteration, wherein when the first iteration is performed for the first time, the point cloud corresponding to the start of the first iteration is determined based on the point cloud collected by the camera after the projector projects a stripe image onto the object and captures the stripe image projected onto the object; based on the point cloud corresponding to the start of the first iteration, a translation vector is calculated; in response to the robot translating according to the translation vector, the point cloud after the robot translates is collected; fuse the point cloud corresponding to the beginning of the first iteration this time with the point cloud collected after the robot translation, to obtain a fused point cloud; in response to the absence of a saturated boundary in the fused point cloud, take the fused point cloud as the point cloud corresponding to the face; in response to the presence of a saturated boundary in the fused point cloud, take the fused point cloud as the point cloud corresponding to the beginning of the first iteration next time; The three-dimensional measurement device of the strongly reflective object further includes: The density clustering module is configured to, for each of the other faces of the object except the preset face, perform density clustering on the saturated area boundary point cloud in the plurality of point clouds, to obtain a plurality of point cloud clusters, wherein the plurality of point clouds include all point clouds acquired and collected in each first iteration process for each face before the current face; The viewpoint calculation module is configured to calculate a viewpoint corresponding to each point cloud cluster in the plurality of point cloud clusters, to obtain a plurality of viewpoints; The viewpoint sorting module is configured to sort the plurality of viewpoints in descending order of the density of the plurality of point cloud clusters, to obtain a sorting result; The second iteration module is configured to, for each of the other faces, calculate the point cloud corresponding to the beginning of the first iteration when the first iteration is initially performed by means of second iteration. For each second iteration process, the second iteration module is configured to perform the following operations until the second iteration terminates: determine the viewpoint at the beginning of the second iteration this time, and collect a point cloud at the viewpoint, wherein the viewpoint at the beginning of the initial second iteration is the first viewpoint in the sorting result; calculate a first distance between the center of the point cloud collected in the second iteration this time and the center of a preset fused point cloud, wherein the preset fused point cloud refers to the fused point cloud obtained by the last first iteration for a face before the current face; in response to the first distance being greater than or equal to a preset threshold, take the point cloud collected in the second iteration this time as the point cloud corresponding to the beginning of the first iteration when the first iteration is initially performed for the current face; in response to the first distance being less than the preset threshold, select the next viewpoint adjacent to the viewpoint at the beginning of the second iteration this time from the sorting result as the viewpoint at the beginning of the next second iteration.

7. An electronic device, comprising: comprise: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the three-dimensional measurement method of the strongly reflective object according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device can perform the three-dimensional measurement method of the strongly reflective object according to any one of claims 1 to 5.

9. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the three-dimensional measurement method of the strongly reflective object according to any one of claims 1 to 5.

Images