Three-dimensional measurement method for strong reflective object and related equipment

Abstract

The invention relates to a three-dimensional measurement method and related equipment for a highly reflective object, and the method comprises the steps: generating a three-dimensional point cloud of an object based on a point cloud corresponding to each surface in a plurality of surfaces; wherein for each surface, a point cloud corresponding to the surface is obtained through a first iteration mode; for each first iteration process, executing the following operations: calculating a translation vector based on the corresponding point cloud when the first iteration starts; collecting a point cloud after translation of the robot; fusing the corresponding point cloud at the beginning of the first iteration with the point cloud acquired after the robot translates; taking the fused point cloud as a point cloud corresponding to the face in response to the fact that no saturated boundary exists in the fused point cloud; and otherwise, taking the fused point cloud as the corresponding point cloud when the next first iteration starts. By controlling the robot to actively move, the saturation area on the object can be autonomously avoided, it can be ensured that the accurate and complete point cloud is obtained, and the accuracy of three-dimensional measurement is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of structured light vision three-dimensional measurement, and more particularly, to a three-dimensional measurement method for strongly reflective objects and related equipment. BACKGROUND

[0002] Robotic autonomous intelligent operation can be applied in many fields, such as machining and grinding, navigation and positioning, reverse engineering, clamping and assembly, defect detection, etc., and obtaining complete and accurate three-dimensional information of a target object is the core of realizing robotic autonomous intelligent operation. At present, three-dimensional measurement of an object is mainly performed by actively projecting a coded pattern onto the object to be measured through structured light vision technology. However, the robustness and completeness of the object point cloud obtained by the robot depend largely on the physical properties of the object surface. For objects with strong reflectivity and no texture, such as machined metal parts, smooth plastics or painted surfaces, the light beam projected by the structured light system is easily saturated in the camera that collects the stripe modulation information after being reflected on the surface of such objects, which makes it impossible to calculate the height information of the corresponding area of the object, resulting in the loss of a large area of point cloud, and further resulting in the difficulty in accurately obtaining the three-dimensional information of such objects.

[0003] In related technologies, in order to realize three-dimensional reconstruction of strongly reflective and textureless objects, researchers have proposed a series of high dynamic range structured light technologies, mainly including: changing the camera exposure parameters, changing the projection brightness, using additional hardware devices, etc. However, the method of changing the camera exposure parameters is difficult to realize pixel-level brightness control and the selection of the best exposure parameter is difficult. The method of changing the projection brightness needs to project a large number of stripe sequences with different brightness to obtain an adaptive projection pattern, which has the disadvantage of low measurement efficiency. Although the use of adaptive local brightness adjustment method can reduce the number of projections, it is still difficult to reliably map the saturated area in the camera image to the projection plane. More importantly, in areas with very high reflectivity, even if the smallest projection intensity is used to suppress the reflection of the strongly reflective area, the corresponding pixels in the center of these shiny areas in the camera-acquired image will still become saturated, resulting in the loss of three-dimensional information. The method of using additional hardware devices usually uses a polarized camera or a light field camera to replace the traditional camera in the structured light system, which reduces the image resolution and increases the system complexity. Therefore, there is an urgent need for a method to measure the three-dimensional information of objects with strong reflectivity and no texture to obtain more accurate three-dimensional information. SUMMARY

[0004] The present disclosure provides a three-dimensional measurement method for strongly reflective objects and related equipment to at least solve the problem that it is difficult to accurately obtain the three-dimensional information of objects with strong reflectivity and no texture in the related art.

[0005] According to a first aspect of embodiments of the present disclosure, a three-dimensional measurement method of a strong reflective object is provided, which is applied to a structured light system, the structured light system comprising a robot, a projector and a camera, the projector and the camera being installed at the end of a mechanical arm of the robot, and the method comprising: acquiring a point cloud corresponding to each of a plurality of surfaces of an object; generating a three-dimensional point cloud of the object based on the point cloud corresponding to each of the plurality of surfaces; wherein for each surface, the point cloud corresponding to the surface is acquired through a first iteration; for each first iteration process, the following operations are performed until the first iteration terminates: acquiring a point cloud corresponding to the start of the current first iteration, wherein when the first iteration is performed for the first time, the point cloud corresponding to the start of the first iteration is determined based on a point cloud collected by the camera for capturing a stripe image projected onto the object after the stripe image is projected onto the object by the projector; calculating a translation vector based on the point cloud corresponding to the start of the current first iteration; in response to the robot translating according to the translation vector, collecting a point cloud after the robot translates; fusing the point cloud corresponding to the start of the current first iteration and the point cloud collected after the robot translates to obtain a fused point cloud; in response to the fused point cloud not having a saturated boundary, taking the fused point cloud as the point cloud corresponding to the surface; in response to the fused point cloud having a saturated boundary, taking the fused point cloud as a point cloud corresponding to the start of the next first iteration.

[0006] Optionally, for a preset surface of the object, when the first iteration is performed for the first time, the point cloud corresponding to the start of the first iteration is a point cloud collected from a preset initial viewpoint.

[0007] Optionally, the three-dimensional measurement method further comprises: for each of the other faces of the object in addition to the preset face, performing density clustering on the saturated region boundary point cloud in the plurality of point clouds to obtain a plurality of point cloud clusters, wherein the plurality of point clouds comprise all point clouds acquired and collected in each first iteration process for each face before the current face; calculating a corresponding view point of each point cloud cluster in the plurality of point cloud clusters to obtain a plurality of view points; sorting the plurality of view points in descending order of density of the plurality of point cloud clusters to obtain a sorting result; for each of the other faces, calculating the point cloud corresponding to the start of the first iteration when the first iteration is performed for the first time by means of a second iteration; wherein for each second iteration process, the following operations are performed until the second iteration terminates: determining the view point at the start of the second iteration and collecting a point cloud at the view point, wherein the view point at the start of the first second iteration is the first view point in the sorting result; calculating a first distance between the center of the point cloud collected in the second iteration and the center of a preset fusion point cloud, wherein the preset fusion point cloud refers to the fusion point cloud obtained by the last first iteration for a face before the current face; in response to the first distance being greater than or equal to a preset threshold, collecting the point cloud in the second iteration as the point cloud corresponding to the start of the first iteration when the first iteration is performed for the first time for the current face; in response to the first distance being less than the preset threshold, selecting the next view point adjacent to the view point at the start of the second iteration from the sorting result as the view point at the start of the next second iteration.

[0008] Optionally, the calculating the view point corresponding to each point cloud cluster in the plurality of point cloud clusters comprises: constructing a reference coordinate system based on the each point cloud cluster, wherein the center of the point cloud cluster is the origin of the reference coordinate system, the normal vector of the point cloud cluster is the z-axis of the reference coordinate system, the principal direction obtained by performing principal component analysis on the point cloud cluster is the y-axis of the reference coordinate system, and the secondary direction obtained is the x-axis of the reference coordinate system; and calculating the view point corresponding to the point cloud cluster based on the reference coordinate system.

[0009] Optionally, the three-dimensional measurement method further comprises: for each first iteration process, further performing the following operations: determining a second distance between the center of the point cloud corresponding to the beginning of the current first iteration and the plane of the projector; calculating a rotation angle based on the second distance; performing cross multiplication between the translation vector and the principal axis vector of the projector to obtain a rotation axis; calculating a rotation matrix based on the rotation axis and the rotation angle; in response to the robot rotating according to the rotation matrix, collecting a point cloud after the robot rotates; wherein the fusion of the point cloud corresponding to the beginning of the current first iteration and the point cloud collected after the robot translates comprises: fusing the point cloud corresponding to the beginning of the current first iteration, the point cloud collected after the robot translates, and the point cloud collected after the robot rotates to obtain the fused point cloud.

[0010] Optionally, the calculation of the translation vector based on the point cloud corresponding to the beginning of the current first iteration comprises: extracting a plurality of saturated region boundary point clouds from the point cloud corresponding to the beginning of the current first iteration; calculating a three-dimensional directional bounding box of each saturated region boundary point cloud in the plurality of saturated region boundary point clouds and determining a vector of the shortest side of the three-dimensional directional bounding box; and calculating the translation vector based on the plurality of shortest side vectors corresponding to the plurality of saturated region boundary point clouds.

[0011] According to a second aspect of the embodiments of the present disclosure, a three-dimensional measurement device for a strongly reflective object is provided, which is applied to a structured light system, the structured light system comprising a robot, a projector and a camera, the projector and the camera being installed at the end of a mechanical arm of the robot, and the device comprising: a single-face point cloud acquisition module configured to acquire a point cloud corresponding to each face of a plurality of faces of an object; and a three-dimensional point cloud generation module configured to generate a three-dimensional point cloud of the object based on the point cloud corresponding to each face of the plurality of faces; wherein for each face, the single-face point cloud acquisition module is configured to acquire the point cloud corresponding to the face through a first iteration; and for each first iteration process, the following operations are performed until the first iteration is terminated: acquiring a point cloud corresponding to the start of the current first iteration, wherein when the first iteration is performed for the first time, the point cloud corresponding to the start of the first iteration is determined based on a point cloud acquired by the camera after a stripe image is projected onto the object by the projector and the stripe image projected onto the object is photographed; calculating a translation vector based on the point cloud corresponding to the start of the current first iteration; acquiring a point cloud after the robot is translated according to the translation vector; fusing the point cloud corresponding to the start of the current first iteration and the point cloud acquired after the robot is translated to obtain a fused point cloud; in response to the absence of a saturated boundary in the fused point cloud, taking the fused point cloud as the point cloud corresponding to the face; and in response to the presence of a saturated boundary in the fused point cloud, taking the fused point cloud as a point cloud corresponding to the start of the next first iteration.

[0012] Optionally, for a preset face of the object, when the first iteration is performed for the first time, the point cloud corresponding to the start of the first iteration is a point cloud acquired from a preset initial viewpoint.

[0013] Optionally, the three-dimensional measuring device of the strong light-reflecting object further comprises: a density clustering module configured to, for each of the other faces of the object except the preset face, perform density clustering on the saturated region boundary point cloud in the plurality of point clouds to obtain a plurality of point cloud clusters, wherein the plurality of point clouds comprise all point clouds acquired and collected in each first iteration process for each face before the current face; a view point calculation module configured to calculate a view point corresponding to each point cloud cluster in the plurality of point cloud clusters to obtain a plurality of view points; a view point sorting module configured to sort the plurality of view points in descending order of density of the plurality of point cloud clusters to obtain a sorting result; and a second iteration module configured to, for each of the other faces, calculate the point cloud corresponding to the start of the first iteration when the first iteration is performed for the first time by means of a second iteration; wherein, for each second iteration process, the second iteration module is configured to perform the following operations until the second iteration terminates: determining a view point at the start of the current second iteration and collecting a point cloud at the view point, wherein the view point at the start of the first second iteration is the first view point in the sorting result; calculating a first distance between the center of the point cloud collected in the current second iteration and the center of a preset fusion point cloud, wherein the preset fusion point cloud refers to a fusion point cloud obtained by the last first iteration for a face before the current face; in response to the first distance being greater than or equal to a preset threshold, taking the point cloud collected in the current second iteration as the point cloud corresponding to the start of the first iteration when the first iteration is performed for the first time for the current face; and in response to the first distance being less than the preset threshold, selecting a next view point adjacent to the view point at the start of the current second iteration from the sorting result as a view point at the start of the next second iteration.

[0014] Optionally, the view point calculation module is configured to: based on the each point cloud cluster, construct a reference coordinate system, wherein the center of the point cloud cluster is the origin of the reference coordinate system, the normal vector of the point cloud cluster is the z-axis of the reference coordinate system, the principal direction obtained by performing principal component analysis on the point cloud cluster is the y-axis of the reference coordinate system, and the secondary direction obtained is the x-axis of the reference coordinate system; and based on the reference coordinate system, calculate the view point corresponding to the point cloud cluster.

[0015] Optionally, the three-dimensional measurement device of the strong light-reflecting object further comprises: a distance determination module configured to determine a second distance between a center of the point cloud corresponding to the start of the first iteration and a plane of the projector; a rotation angle calculation module configured to calculate a rotation angle based on the second distance; a cross product module configured to cross multiply the translation vector and a principal axis vector of the projector to obtain a rotation axis; a rotation matrix calculation module configured to calculate a rotation matrix based on the rotation axis and the rotation angle; and a rotated point cloud acquisition module configured to acquire a point cloud after the robot is rotated according to the rotation matrix. The single-surface point cloud acquisition module is configured to fuse the point cloud corresponding to the start of the first iteration, the point cloud acquired after the robot is translated, and the point cloud acquired after the robot is rotated to obtain the fused point cloud.

[0016] Optionally, the single-surface point cloud acquisition module is configured to extract a plurality of saturated region boundary point clouds from the point cloud corresponding to the start of the first iteration; calculate a three-dimensional directional bounding box of each saturated region boundary point cloud in the plurality of saturated region boundary point clouds, and determine a vector of a shortest side of the three-dimensional directional bounding box; and calculate the translation vector based on the plurality of vectors of the shortest sides corresponding to the plurality of saturated region boundary point clouds.

[0017] According to a third aspect of embodiments of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a three-dimensional measurement method of a strong light-reflecting object according to the present disclosure.

[0018] According to a fourth aspect of embodiments of the present disclosure, a computer-readable storage medium is provided, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform a three-dimensional measurement method of a strong light-reflecting object according to the present disclosure.

[0019] According to a fifth aspect of embodiments of the present disclosure, a computer program product is provided, comprising a computer program, when the computer program is executed by a processor, a three-dimensional measurement method of a strong light-reflecting object according to the present disclosure is implemented.

[0020] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:

[0021] In the present disclosure, for each of the plurality of faces contained in the object, the point cloud of the face can be calculated by controlling the robot to move multiple times in an iterative manner, i.e., the present disclosure can control the robot to perform active motion to autonomously avoid the saturated area, i.e., the strong reflection area, on the object. In this way, accurate and complete point clouds can be ensured, and thus the three-dimensional information of the measured object can be ensured to be consistent with the actual situation of the object, and the accuracy of the three-dimensional measurement of the strong reflection and textureless object is improved.

[0022] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0024] Figure 1 is a schematic diagram showing a structured light system in an actual measurement scene according to an exemplary embodiment of the present disclosure;

[0025] Figure 2 is a schematic diagram showing a plurality of modules contained in a structured light system according to an exemplary embodiment of the present disclosure;

[0026] Figure 3 is a flowchart showing a three-dimensional measurement method of a strong reflection object according to an exemplary embodiment of the present disclosure;

[0027] Figure 4 is a schematic diagram showing optical modeling of a measurement scene according to an exemplary embodiment of the present disclosure;

[0028] Figure 5 is a schematic diagram showing an adaptive structured light process according to an exemplary embodiment of the present disclosure;

[0029] Figure 6 is a schematic diagram showing active motion of a robot to avoid a strong reflection area according to an exemplary embodiment of the present disclosure;

[0030] Figure 7 is a schematic diagram showing robot viewpoint generation according to an exemplary embodiment of the present disclosure;

[0031] Figure 8 is a schematic diagram showing a measured object according to an exemplary embodiment of the present disclosure;

[0032] Figure 9 is a schematic diagram showing measurement results obtained by performing three-dimensional measurement on a strong reflection and textureless object according to an exemplary embodiment of the present disclosure;

[0033] Figure 10 is a block diagram of a three-dimensional measurement device for a strongly light-reflecting object according to an example embodiment of the present disclosure;

[0034] Figure 11 is a block diagram of an electronic device according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] In order to make the ordinary person skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings.

[0036] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. The embodiments described in the following embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0037] It should be noted herein that "at least one of a plurality of items" appearing in the present disclosure means that the three types of alternatives are included: "any one of the plurality of items", "a combination of any two or more of the plurality of items", and "all of the plurality of items". For example, "including at least one of A and B" includes the following three alternatives: (1) including A; (2) including B; and (3) including A and B. For another example, "performing at least one of step one and step two" means the following three alternatives: (1) performing step one; (2) performing step two; and (3) performing step one and step two.

[0038] Figure 1 is a schematic diagram of a structured light system in an actual measurement scene according to an example embodiment of the present disclosure.

[0039] Referring to Figure 1The structured light system can mainly include a structured light sensor device and a robot. The structured light sensor device can mainly include a projector and an industrial camera. The projector is mainly used to project a coded fringe image to a target object. The industrial camera is mainly used to capture a modulated coded fringe image, i.e., the industrial camera is mainly used to capture a coded fringe image reflected by the surface of the target object. The robot is mainly used to carry the structured light sensor device to perform active motion to avoid a strong reflection area on the surface of the target object, so that a complete three-dimensional point cloud of the target object can be constructed based on point clouds captured at multiple viewpoints during the active motion.

[0040] Figure 2 FIG. 1 is a schematic diagram showing a plurality of modules included in a structured light system according to an example embodiment of the present disclosure.

[0041] Referring to FIG. 1 Figure 2 The structured light system can mainly include a "measurement scene optical modeling module", an "adaptive structured light module", a "robot strong reflection avoidance module", and a "robot viewpoint planning module". Sequentially executing the above modules can achieve accurate three-dimensional reconstruction of a strong reflection and a textureless object surface.

[0042] The "measurement scene optical modeling module" can model a structured light measurement scene by using a physics-based rendering technique, and can then obtain the influence of reflectivity of a surface of a target object on camera saturation from an energy perspective, thereby providing a theoretical basis for the adaptive structured light module.

[0043] The "adaptive structured light module" can mainly include three parts: "adaptive projection brightness generation", "saturation area matching", and "three-dimensional point cloud generation". The function of the adaptive structured light module is mainly to obtain a local point cloud of a high-reflectivity object surface. Specifically, the structured light system can project a coded fringe and a uniform brightness sequence to an object surface, and can trigger a camera to synchronously capture and store an image reflected by the object surface.

[0044] The "adaptive projection intensity generation" can be implemented based on a convolutional neural network, wherein the input of the network can be an image captured under a uniform intensity projection sequence, and the output can be an optimal projection intensity map of the camera plane; the "saturation region matching" can be based on the correspondence between the edge pixels of the saturation region in the image and the corresponding projector position, to calculate the homography relationship between the saturation region of the camera image and the corresponding region of the projector projection plane, and then the optimal projection intensity map of the projector plane can be obtained; the "three-dimensional point cloud generation" can decode the adaptive coded image modulated by the object surface height collected by the camera, and then the matching relationship of the camera image and the projector image pixels can be obtained. Furthermore, by combining the internal and external parameters of the camera and the projector, the three-dimensional point cloud of the measured object surface can be calculated.

[0045] The "robot strong reflection avoidance module" can mainly include three parts of "translation motion", "rotation motion" and "compensation motion", and its function is mainly to obtain the complete local point cloud of the high-reflective object surface. Specifically:

[0046] The translation motion can be that the structured light sensor device translates along the object surface with the minimum displacement to accurately avoid the original strong reflection region, so as to ensure that the missing point cloud region is adjacent but not overlapped. Furthermore, since the translation motion changes the field of view of the camera, the consistency of the three-dimensional measurement region can be maximally maintained through the rotation motion. In addition, since the rotation motion changes the posture of the projector, it will change the incidence angle of the light beam on the measurement surface, and then it will cause the change of the reflection position of the object surface and the position of the saturated region in the image, thereby reducing the accuracy of avoiding the strong reflection region. Therefore, the displacement of the reflection region in the image can also be compensated by additional translation motion.

[0047] The "robot viewpoint planning module" can mainly include "viewpoint generation" and "viewpoint adjustment". Specifically: the viewpoint generation can be to predict the next observation posture of the robot according to the existing point cloud information, so as to realize the effective exploration of the unknown region of the measured object; the viewpoint adjustment can be to allow the selection of a suboptimal viewpoint when the generated viewpoints are almost in the same position to ensure the continuation of the measurement process.

[0048] The present disclosure provides an improved scheme from hardware to algorithm to realize the three-dimensional reconstruction of strong reflection and textureless object surface. The scheme integrates theoretical optical modeling, adaptive structured light algorithm, robot-assisted strong reflection active avoidance and point cloud-based viewpoint planning, realizes the accurate, robust and complete three-dimensional reconstruction of large-size, strong-reflective and textureless objects, and has strong practicality and engineering value.

[0049] Figure 3is a flowchart illustrating a three-dimensional measurement method of a strongly light-reflecting object according to an example embodiment of the present disclosure, applied to a structured light system. As previously described, the structured light system can include a robot, a projector, and a camera, and the projector and the camera can be mounted at the end of a robot arm of the robot.

[0050] Referring to Figure 3 In step 301, a point cloud corresponding to each of a plurality of faces of an object can be acquired. For example, assuming that an object to be measured has six faces, a point cloud corresponding to each of the six faces of the object can be acquired.

[0051] In step 302, a three-dimensional point cloud of the object can be generated based on the point cloud corresponding to each of the plurality of faces, i.e., a surface profile of the object to be measured can be three-dimensionally reconstructed based on the point cloud corresponding to each of the plurality of faces included in the object to be measured.

[0052] Specifically, for each face, the point cloud corresponding to the face can be acquired through a first iteration.

[0053] And for each first iteration process, the following operations can be performed until the first iteration is terminated:

[0054] First, a point cloud corresponding to the start of the current first iteration can be acquired, wherein when the first iteration is performed for the first time, the point cloud corresponding to the start of the first iteration can be determined based on the point cloud acquired by the camera after the projector projects a stripe image onto the object and photographs the stripe image projected onto the object.

[0055] Then, a translation vector can be calculated based on the point cloud corresponding to the start of the current first iteration. Next, in response to the robot translating according to the calculated translation vector, a point cloud after the robot translates can be acquired. Then, the point cloud corresponding to the start of the current first iteration and the point cloud acquired after the robot translates can be fused to obtain a fused point cloud.

[0056] In response to the absence of a saturated boundary in the fused point cloud, the fused point cloud can be used as the point cloud corresponding to the face; in response to the presence of a saturated boundary in the fused point cloud, the fused point cloud can be used as the point cloud corresponding to the start of the next first iteration.

[0057] It should be noted that the "saturation boundary" is the "strong reflection region boundary", which can be regarded as a "hole" in the fused point cloud. If there is no saturation boundary in the fused point cloud, it means that there is no strong reflection region in the fused point cloud, that is, there is no "hole" in the fused point cloud, at this time, it means that the complete point cloud information of the corresponding surface has been obtained. Therefore, the fused point cloud can be directly used as the point cloud corresponding to the surface. If there is a saturation boundary in the fused point cloud, it means that there is still a strong reflection region in the fused point cloud, that is, there is still a "hole" in the fused point cloud, at this time, it means that the complete point cloud information of the corresponding surface has not been obtained. Therefore, the fused point cloud can be used as the point cloud corresponding to the surface at the beginning of the next first iteration of the current first iteration to continue the iteration process until the iteration is terminated.

[0058] Figure 4 is a schematic diagram showing the measurement scene optical modeling according to an example embodiment of the present disclosure. Referring to Figure 4 The light energy received by each image pixel can be discretely modeled. It should be noted that the intensity value of each image pixel is proportional to the light energy reflected to the camera at the pixel. And considering factors such as ambient light, object surface mutual reflection light, camera noise, camera gain and exposure time, a complete imaging model can be obtained. Further, the brightness of the projector projection pattern can also be changed uniformly and the camera can acquire images. Then, based on the imaging model, the reflectivity of each image pixel area can be estimated using least squares fitting.

[0059] It should be noted that each time the projector projects a fringe image onto the object, two projections can be performed. The first projection is mainly used to obtain the reflectivity of each position on the object, which can be used to guide the adjustment of the projection brightness of the second projection of the fringe image by the projector. For example, the higher the reflectivity, the lower the brightness of the second projection; the lower the reflectivity, the higher the brightness of the second projection, and so on.

[0060] In addition, referring to (b) in Figure 4 In the light projection-receiving system, the propagation of light follows the principles of geometric optics. The object surface position p can be modeled using the microplane theory, that is, it can be represented as a collection of many small ideal reflecting planes with random directions at a microscopic scale. And in (c) in Figure 4 ref L dif is the diffuse reflection light intensity, L absor is the light intensity absorbed by the workpiece surface, L spe is the specular reflection light intensity, represents the normal vector at position p. In the structured light system, referring to Figure 4 ​(c) in the equation (1), the micro-surface is defined in a spatial region corresponding to a single pixel. The light energy received by each pixel can be discretely modeled, where the value at (x c ,y c ) is proportional to the light energy reflected to the camera:

[0061]

[0062] where the intensity of the incident radiation in the direction at position p is The reflected intensity in the direction is denoted as K g , a constant representing the mapping relationship between the light energy and the pixel value, and α is the reflectivity,

[0063] (x c ,y c ) is the image coordinate, I(x c ,y c ) is the pixel intensity value of the point (x c ,y c ), and its value range can be 0-255, 0 representing complete black and 255 representing complete white.

[0064] In addition, considering the ambient light component and the inter-reflected light L inter from the object surface, the camera noise I noise , the camera gain k, and the exposure time t, etc., a complete camera imaging model can also be derived:

[0065]

[0066] where L in includes and L inter (p), where is the intensity of the incident projector light in the direction at position p, is the intensity of the incident ambient light in the direction at position p, and L inter (p) is the intensity of the incident inter-reflected light from the workpiece surface at position p.

[0067] Then, there can be:

[0068] I(x c ,y c ) = αx1+ βx2+ I noise

[0069] where:

[0070] And by collecting n sets of data (I i ,x i1 ,x i2 ) for each pixel while systematically varying the intensity of the projector, one can estimate a, b using least squares, specifically one can define Q as:

[0071]

[0072] To optimize Q, one can set the partial derivatives with respect to a and b to zero, resulting in:

[0073]

[0074] And one can define X = [x 11 x 12 ,x 21 x 22 ,…,x n1 x n2 ], B = [a, b] T , I = [I1, I2, …, I n ] T , and one can then rewrite the above equation as X T XB = X T I.

[0075] If X T X is invertible, then the estimates of B and I can be obtained by:

[0076]

[0077] In (d) of Figure 4 , represents the normal vector at the infinitesimal p, dA represents the infinitesimal area in calculus, w Ψ represents the direction of the incident light.

[0078] The key advantage of this analysis is that it can model the reflectance properties of the target. Specifically, by defining a surface reflectance a and optically modeling the structured light measurement of the scene, one can effectively describe the surface properties of the object. Note that the surface reflectance parameter a is a fundamental component that controls the projection behavior of the structured light system in highly reflective regions, and it can also provide a basis for the motion planning of the robot to mitigate strong reflections.

[0079] Figure 5 is a schematic diagram showing an adaptive structured light process according to an example embodiment of the present disclosure. Referring to (a) of Figure 5 and (b) of Figure 5 , the actively controllable parameters that have a large impact on the pixel values are k, t and L proj. k and t can be fixed as constants k fix and t fix , and the projection intensity can be adjusted to reach the ideal unsaturated pixel value I exp . In this way, the optimal projection intensity I

[0080]

[0081] Once I exp is set, I can be calculated on a per-pixel basis Figure 5 However, when the pixel is large, performing matrix inversion on each pixel will result in excessive computation. Therefore, a method based on convolutional neural network can be introduced, which can take the image captured under the uniform intensity projection sequence as input, and then quickly generate the optimal projection intensity map. Specifically, referring to (c) in , the network architecture can consist of two branches, namely the first branch (branch 1), i.e. the main branch and the second branch (branch 2). The main branch (branch 1) can adopt a multi-scale high-resolution feature extraction backbone paired with a self-attention module, and can process multi-channel input images and identify saturated regions. Then, multi-class classification can be performed on these pixel positions to assign appropriate intensity categories. The second branch (branch 2) can extract features from the input image and predict specific intensity values for each pixel class. Finally, the outputs of the aforementioned two branches can be integrated to generate the optimal projection intensity map

[0082] It should be noted that is defined in the camera imaging plane, and in order to effectively reduce the reflection of the target surface, it is necessary to convert to This conversion can be achieved by calculating the homography of each saturated region to ensure accurate mapping of the optimal projection intensity. Specifically, to achieve this, the projector can be modeled as an inverse camera, and it can be assumed that both the camera and the projector follow the pinhole imaging model.

[0083] s c (x c ,y c ,1) T =K c [R c ∣t c ](X w ,Y w ,Z w ) T

[0084] sp (x p ,y p ,1) T =K p [R p ∣t p ](X w ,Y w ,Z w ) T .

[0085] In this model, s c ,s p are scale factors, K c ,K p are intrinsic parameters of the camera and the projector, [R c ∣t c ] and [R p ∣t p ] are extrinsic parameters of the camera and the projector. If the vertical direction stripes are known, i.e. the x p direction code values are known, i.e. x p is known, then for a point (x c ,y c ) in the camera plane, the goal is to find the corresponding y p in the projector plane to form a point pair relationship, by decoding the code values corresponding to x p .

[0086]

[0087] And (X w ,Y w ,Z w ) can be taken as intermediate values in the calculation process, and then the following can be solved:

[0088]

[0089] Now only s p and y p are unknown, but two equations can be constructed to solve these two unknowns respectively:

[0090]

[0091] At this time, the corresponding relationship between the edge pixels (x c ,y c ) in the saturated area of the image and the corresponding projector positions (x p ,y p ) can be established, and the homography relationship from the camera imaging plane to the saturated area of the projector projection plane can be constructed:

[0092] (x p ,y p ,1) T =H·(x c ,y c ,1) T

[0093] By identifying four non-collinear point pairs along the boundary of the saturated region, the matrix H can be solved. Finally, by mapping from (x c ,y c ) to (x p ,y p ), the corresponding local dark fringe pattern can be generated, the specific calculation process referring to (d) in Figure 5 . Next, referring to (e) in Figure 5 , the projection can be performed again according to the optimized brightness mode.

[0094] In addition, in order to enhance the robustness, an encoding scheme integrating gray code and line shift code can also be used, which can minimize the impact of reflection in the initial encoding mode design and decoding process. In addition, sub-pixel edge extraction can also be performed using the fringe, which can further improve the accuracy of the point cloud.

[0095] It should be noted that the above framework is tailored for the measurement scene of high-reflective targets, which comprehensively considers the selection of encoding mode, decoding, triangulation algorithm, efficient calculation of optimal projection intensity, and accurate mapping of saturated regions, etc. The main goal is to mitigate the adverse effects of high reflection on measurement accuracy at each algorithm stage.

[0096] Figure 6 is a schematic diagram showing the robot active motion to avoid the strong reflection region according to an example embodiment of the present disclosure.

[0097] Referring to Figure 6 , as described before, the robot strong reflection avoidance module can mainly include translational motion, rotational motion and compensation motion. The light beam of the projector produces strong reflection at the X a position, causing saturation near the pixel position of the captured image. Then, the motion of the sensor moves the corresponding reflection region on the object from X a to X b . Since the field of view of the camera changes after moving, rotation is needed to ensure that the field of view remains essentially unchanged. After rotation, the saturated region is X c , which causes saturation near the pixel position of the captured image. Finally, compensation motion can be performed, which causes saturation near the pixel position of the captured image, specifically:

[0098] Referring to Figure 6 (b), the translational motion can be a translation of the structured light sensor along the object surface with minimum displacement to precisely avoid the original strong reflection region, thus ensuring the missing point cloud region is adjacent without overlapping. Also, referring to Figure 6 (c), since the translational motion changes the field of view of the camera, the consistency of the three-dimensional measurement region can also be maximized by the rotational motion. In addition, referring to Figure 6 (d), since the rotational motion changes the pose of the projector, which changes the incident angle of the light beam on the measurement surface, and further causes the change of the object surface reflection position and the saturated region position in the image, resulting in the decrease of the accuracy of avoiding the strong reflection region. Therefore, the displacement of the reflection region in the image can also be compensated by additional translational motion.

[0099] According to the exemplary embodiments of the present disclosure, a plurality of saturated region boundary point clouds can be extracted from the corresponding point cloud at the beginning of the first iteration of this time. Then, a three-dimensional oriented bounding box of each of the plurality of saturated region boundary point clouds can be calculated, and a vector of the shortest side of the plurality of sides contained in the three-dimensional oriented bounding box can be determined. Further, the surface area and the center of the three-dimensional oriented bounding box can also be determined. Next, a translation vector can be calculated based on the plurality of shortest side vectors corresponding to the plurality of saturated region boundary point clouds one by one. Specifically, the weighted value of all the shortest side vectors can be taken as the direction vector of the translational motion, and the projection value of the largest shortest side vector in the direction of the translational motion can be taken as the translational motion amount.

[0100] In this way, the robot strong reflection avoidance module can stop the saturation of the previously saturated surface region in the image through translational motion, and further allow the missing point cloud to be recovered through additional measurement. Specifically, in order to minimize the moving distance, the point cloud collected at the current viewpoint can be analyzed. For example, first, the boundaries of all saturated regions can be extracted from the fringe image, and the point cloud corresponding to each saturated region boundary can be calculated Then, for each a three-dimensional oriented bounding box (OBB) can be calculated, and further the surface area center and the shortest side vector of each OBB can be obtained. Next, the weighted unit vector obtained by weighting all the shortest side vectors can be taken as the moving direction of the sensor. Finally, the projection value of the largest shortest side vector in the direction of the translational motion

[0101]

[0102] According to an exemplary embodiment of the present disclosure, for each first iteration process, the following operations may be further performed:

[0103] Determine the second distance between the center of the point cloud corresponding to the start of this first iteration and the plane of the projector. Then, the rotation angle can be calculated based on the second distance. Next, the translation vector can be cross-multiplied with the principal axis vector of the projector to obtain the rotation axis. Then, the rotation matrix can be calculated based on the rotation axis and the rotation angle. Exemplarily, the rotation matrix can be constructed using the Rodrigues rotation formula. Next, in response to the robot rotating according to the rotation matrix, the point cloud after the robot rotates can be collected. Then, the point cloud corresponding to the start of this first iteration, the point cloud collected after the robot translates, and the point cloud collected after the robot rotates can be fused to obtain a fused point cloud.

[0104] It should be noted that the robot's strong reflection avoidance translation movement will change the camera's field of view and the projector's illumination range at the current viewpoint. Therefore, it is necessary to maximize the consistency of the three-dimensional measurement area through the robot's rotation movement. Specifically, assume that the rotation center of the robot's end effector before and after the translation movement is recorded as O and O' respectively, and the distance from O to the projector is R, and the point cloud center C ptc The distance to the projection plane before motion is D. Once the projector to camera transformation projection is calibrated and the hand-eye matrix The spatial geometric relationship between the robot and the sensor and the relative relationship between their coordinate systems can be directly determined. Specifically, the robot's rotation angle θ can be calculated by establishing a reference plane rot , and the normal of the reference plane can be determined by the cross product of the projector axis vector and the translation motion vector, which can be expressed as: In addition, according to the geometric relationship, we can have:

[0105]

[0106] Among them, α rot is the projector principal axis vector with o′o p The angle between the robot and the end effector is o', which is the rotation center of the robot end effector. p is the optical center of the projector.

[0107] Given a known posture of the robot, the only unknown quantity in the above equation is θ rot , which can be determined numerically. And, in order to obtain the final homogeneous matrix of the rotational motion, the rotation axis and its direction can be defined as: Then, the Rodrigues rotation formula can be applied to construct the rotation matrix R rot :

[0108]

[0109] In addition, the robot's strong reflection avoidance rotation movement changes the posture of the projector, which will change the incident angle of the light beam on the measurement surface, and then the reflection position on the surface will change from X to b Move to X c ,like Figure 6 (c) in the figure. Simultaneously, the camera's imaging plane pose changes, causing saturated areas in the image to shift, which reduces the accuracy of avoiding strongly reflective areas. To address this issue, the robot can also be guided to fine-tune the sensor's viewing posture to compensate for the displacement of reflective areas in the image.

[0110] The compensation action can be performed in two steps. First, after calculating the displacement of the reflective area on the surface, an additional translational motion can be applied to reposition the reflective area to its intended location. The amount of compensation motion can be the amount by which the center of the saturation area is moved due to the translational and rotational motions. The amount of movement of the center of the saturation area caused by translational motion The difference between the sensor and the object can then be adjusted to maintain the standard distance. Specifically, after translation, rotation, and compensation movements, the sensor can be moved along Reposition to keep the point cloud center C ptc and the projection plane Ω proj The distance between them is equal to D, and the adjustment of the distance can be recorded as

[0111] According to an exemplary embodiment of the present disclosure, for a preset face of an object, when the first iteration is initially performed, the corresponding point cloud at the start of the first iteration may be a point cloud collected from a preset initial viewpoint. The preset face may be the first face of the object measured during the three-dimensional measurement process.

[0112] According to an example embodiment of the present disclosure, for each of the other faces of the object other than the preset face, the saturated region boundary point cloud in the plurality of point clouds can be subjected to density clustering to obtain a plurality of point cloud clusters, wherein the plurality of point clouds can include all point clouds obtained and collected in each first iteration process for each face before the current face. For example, assuming that 3 faces of the object have been measured and the 4th face of the object needs to be measured. At this time, all point clouds obtained and collected in each first iteration process for each of the 3 faces that have been measured can be subjected to density clustering. For example, assuming that the 1st face has undergone 3 first iterations to obtain complete point clouds of the face, the 2nd face has undergone 5 first iterations to obtain complete point clouds of the face, and the 3rd face has undergone 4 first iterations to obtain complete point clouds of the face, the point clouds obtained and collected in all first iterations (3+5+4=12 first iterations) corresponding to the 3 faces can be subjected to density clustering.

[0113] Then, the viewpoint corresponding to each of the plurality of point cloud clusters can be calculated to obtain a plurality of viewpoints. For example, assuming that 10 point cloud clusters are obtained in total, 10 viewpoints can be calculated accordingly. Next, the plurality of viewpoints can be sorted in descending order of the density of the plurality of point cloud clusters to obtain a sorting result. For example, the 10 viewpoints can be sorted in descending order of the density of the 10 point cloud clusters to obtain a sorting result.

[0114] It should be noted that for each of the other faces, the point cloud corresponding to the start of the first iteration when the first iteration is initially performed can be calculated by means of the second iteration. Moreover, for each second iteration process, the following operations can be performed until the second iteration terminates:

[0115] Firstly, the viewpoint at the start of the current second iteration can be determined, and a point cloud can be collected at the viewpoint, wherein the viewpoint at the start of the initial second iteration can be the first viewpoint in the aforementioned sorting result. For example, as described above, assuming that 10 viewpoints are obtained in total, the viewpoint at the start of the initial second iteration of the 4th face can be the 1st viewpoint in the aforementioned sorted 10 viewpoints.

[0116] Then, a first distance between the center of the point cloud collected in the current second iteration and the center of a preset fusion point cloud can be calculated, wherein the preset fusion point cloud can refer to the fusion point cloud obtained in the last first iteration for a face before the current face. For example, as described above, the current face can be the 4th face, and the preset fusion point cloud can refer to the fusion point cloud obtained in the last first iteration for the 3rd face.

[0117] Next, in response to the aforementioned first distance being greater than or equal to the preset threshold, the point cloud collected in the second iteration may be used as the point cloud corresponding to the start of the first iteration when the first iteration is initially executed for the current surface. For example, as described above, if the current surface is the fourth surface, the point cloud collected in the second iteration may be used as the point cloud corresponding to the start of the first iteration when the first iteration is initially executed for the fourth surface.

[0118] Optionally, in response to the aforementioned first distance being less than a preset threshold, a next viewpoint adjacent to the viewpoint at the start of the current second iteration may be selected from the sorted results as the viewpoint at the start of the next second iteration. For example, as described above, assuming that a total of 10 viewpoints are obtained, and the viewpoint at the start of the current second iteration is the first of the 10 viewpoints, then if the aforementioned first distance is less than the preset threshold, a second viewpoint adjacent to the first viewpoint at the start of the current second iteration may be selected from the sorted 10 viewpoints as the viewpoint at the start of the next second iteration of the current second iteration, thereby continuing the measurement process. Furthermore, the aforementioned first distance may be, but is not limited to, a Euclidean distance.

[0119] It should be noted that the above description of the second iteration process uses an object with a total of six faces, with the current face being the fourth face, as an example. In practice, the object can have any number of faces, for example, three, eight, or more. Furthermore, the second iteration process for any face other than the first face of the object is similar to the second iteration process for the fourth face described above, and will not be further elaborated here. The above implementation is merely an example.

[0120] As mentioned above, the robot viewpoint generation mainly depends on the representation of point cloud density. Specifically, first, the point cloud of the saturated area boundary can be clustered by density, and the points with the largest number of points (denoted as ) as the first reference cluster generated by the viewpoint. Then, the reference coordinate system can be constructed based on the first reference cluster. Specifically, we can define Center As the coordinate origin, we can define The normal vector is In addition, principal component analysis (PCA) can be applied to extract The main and secondary directions can be assigned as follows: and So we can get the reference coordinate system Representation in the robot reference coordinate system.

[0121] Figure 7 is a schematic diagram illustrating robot viewpoint generation according to an exemplary embodiment of the present disclosure.Figure 7 the projection plane of the projector has its long axis is the major axis of the projector coordinate system, is the minor axis of the projection plane of the projector. And, in order to maximize the coverage of the boundary region, needs to be aligned with the major direction so that the next viewpoint can be constructed for the projector:

[0122]

[0123] One advantage of this viewpoint generation method is that it can ensure sufficient overlap between the point clouds captured at consecutive viewpoints, which facilitates the registration and fusion of multiple point cloud frames in the subsequent process.

[0124] In addition, in the robot viewpoint generation, when the target surface of the object has discontinuous regions or there are occlusions, the new point cloud increment generated by the measurement of these regions is usually insufficient. Furthermore, the boundary regions of the merged point clouds from the previous viewpoint and the current viewpoint can exhibit little update, which can lead to the generation of the subsequent viewpoint at almost the same location, thereby causing the exploration process to be delayed. In order to alleviate this situation, it must be ensured that these situations can be discovered in time and the viewpoint can be adjusted accordingly to ensure continuous and comprehensive observation of the target surface of the object. To this end, the Euclidean distance between the center C i of the merged point cloud obtained in the last first iteration and the center C i-1 of the point cloud collected at the current viewpoint can be calculated as a criterion for whether the updated viewpoint needs to be adjusted. Specifically: if the calculated Euclidean distance is less than a threshold value D th , it indicates that there are special surface conditions that need to trigger the viewpoint adjustment strategy. Exemplarily, the viewpoint adjustment can be to determine the second largest cluster point as the suboptimal viewpoint according to the aforementioned point cloud clustering result and use it as the adjusted viewpoint, so that the measurement process continues.

[0125] According to an exemplary embodiment of the present disclosure, a reference coordinate system can be constructed based on each point cloud cluster. The center of the point cloud cluster can be the origin of the reference coordinate system, and the normal vector of the point cloud cluster can be the z-axis of the reference coordinate system. The major direction obtained by performing principal component analysis on the point cloud cluster can be the y-axis of the reference coordinate system, and the minor direction can be the x-axis of the reference coordinate system. Then, the viewpoint corresponding to the point cloud cluster can be calculated based on the constructed reference coordinate system.

[0126] Figure 8 is a schematic diagram showing a measured object according to an exemplary embodiment of the present disclosure. Referring to Figure 8The measured object can be a strong light reflection machined metal part, a convex pyramid with smooth paint, and a concave bowl. The characteristics of these measured objects are all strong reflectivity and lack of surface texture.

[0127] Figure 9 is a schematic diagram showing measurement results obtained by three-dimensional measurement of a strong reflection, textureless object according to an example embodiment of the present disclosure. Referring to Figure 9 The structured light system provided by the present disclosure has excellent three-dimensional reconstruction performance on all types of surfaces of strong reflection, textureless objects, and the point cloud reconstruction density is also high.

[0128] To solve the problem that it is difficult to accurately perform three-dimensional measurement on strong reflection, textureless objects in the related art, the present disclosure provides a robot active three-dimensional measurement method based on structured light vision. This method mainly has the following advantages:

[0129] (1) The robot can be controlled to actively move to avoid strong reflection areas on the object, and thus the complete local point cloud can be ensured, so that the accumulated local point cloud can be used to plan the viewpoint for comprehensive three-dimensional measurement of the object.

[0130] (2) By setting a robot viewpoint planning module, three-dimensional measurement of large-size, strong light reflection, and textureless objects can be ensured, and the application range is wide.

[0131] (3) The structured light system provided by the present disclosure has simple hardware, which can only include a projector, a camera, and a robot, without any additional hardware devices, and has low cost.

[0132] Figure 10 is a block diagram of a three-dimensional measurement device 1000 for a strong light reflection object according to an example embodiment of the present disclosure. The three-dimensional measurement device 1000 can be applied to a structured light system, which can include a robot, a projector, and a camera. Moreover, the projector and the camera can be installed at the end of the mechanical arm of the robot.

[0133] Referring to Figure 10 The three-dimensional measurement device 1000 can include a single-sided point cloud acquisition module 1001 and a three-dimensional point cloud generation module 1002.

[0134] The single-sided point cloud acquisition module 1001 can acquire a point cloud corresponding to each of a plurality of faces of an object. For example, assuming that the object to be measured has six faces, a point cloud corresponding to each of the six faces of the object can be acquired.

[0135] The three-dimensional point cloud generation module 1002 can generate a three-dimensional point cloud of the object based on the point cloud corresponding to each of the plurality of surfaces, i.e., can perform three-dimensional reconstruction on the surface profile of the object to be measured based on the point cloud corresponding to each of the plurality of surfaces included in the object to be measured.

[0136] For each surface, the single-surface point cloud acquisition module 1001 can acquire the point cloud corresponding to the surface through first iteration. For each first iteration process, the single-surface point cloud acquisition module 1001 can perform the following operations until the first iteration terminates:

[0137] First, the point cloud corresponding to the start of the current first iteration can be acquired. When the first iteration is performed for the first time, the point cloud corresponding to the start of the first iteration can be determined based on the point cloud acquired by the camera after the camera captures the projected fringe image on the object based on the projection of the fringe image on the object by the projector.

[0138] Then, the translation vector can be calculated based on the point cloud corresponding to the start of the current first iteration. Next, in response to the translation of the robot according to the calculated translation vector, the point cloud after the translation of the robot can be acquired. Then, the point cloud corresponding to the start of the current first iteration and the point cloud acquired after the translation of the robot can be fused to obtain a fused point cloud.

[0139] In response to the absence of a saturated boundary in the fused point cloud, the fused point cloud can be taken as the point cloud corresponding to the surface. In response to the presence of a saturated boundary in the fused point cloud, the fused point cloud can be taken as the point cloud corresponding to the start of the next first iteration.

[0140] According to an exemplary embodiment of the present disclosure, the single-surface point cloud acquisition module 1001 can extract a plurality of saturated region boundary point clouds from the point cloud corresponding to the start of the current first iteration. Then, a three-dimensional directional bounding box of each of the plurality of saturated region boundary point clouds can be calculated, and a vector of the shortest side of a plurality of sides included in the three-dimensional directional bounding box can be determined. Further, the surface area and the center of the three-dimensional directional bounding box can also be determined. Next, the translation vector can be calculated based on the plurality of vectors of the shortest sides corresponding to the plurality of saturated region boundary point clouds. Specifically, the weighted value of all the vectors of the shortest sides can be taken as the direction vector of the translation motion, and the projection value of the largest vector of the shortest sides in the direction of the translation motion can be taken as the translation motion amount.

[0141] According to an exemplary embodiment of the present disclosure, the three-dimensional measurement device 1000 described above can further include a distance determination module, a rotation angle calculation module, a cross product module, a rotation matrix calculation module, and a rotated point cloud acquisition module.

[0142] For each first iteration process, the following operations can also be performed:

[0143] The distance determination module is configured to determine a second distance between the center of the point cloud corresponding to the start of the current first iteration and the plane of the projector. Then, the rotation angle calculation module is configured to calculate the rotation angle based on the second distance. Next, the cross product module is configured to obtain the rotation axis by performing a cross product operation on the translation vector and the principal axis vector of the projector. Then, the rotation matrix calculation module is configured to calculate the rotation matrix based on the rotation axis and the rotation angle. Illustratively, the rotation matrix can be constructed using the Rodrigues rotation formula. Next, the rotated point cloud acquisition module can acquire the point cloud after the robot is rotated according to the rotation matrix. Then, the single-sided point cloud acquisition module 1001 can fuse the point cloud corresponding to the start of the current first iteration, the point cloud acquired after the robot is translated, and the point cloud acquired after the robot is rotated to obtain the fused point cloud.

[0144] According to an exemplary embodiment of the present disclosure, for the preset face of the object, the point cloud corresponding to the start of the first iteration when the first iteration is first performed can be the point cloud acquired from the preset initial viewpoint. The preset face can be the first face of the object measured in the three-dimensional measurement process.

[0145] According to an exemplary embodiment of the present disclosure, the three-dimensional measurement device 1000 described above can further include a density clustering module, a viewpoint calculation module, a viewpoint sorting module, and a second iteration module.

[0146] For each of the other faces of the object in addition to the preset face, the density clustering module can perform density clustering on the saturated region boundary point cloud in the plurality of point clouds to obtain a plurality of point cloud clusters, wherein the plurality of point clouds can include all point clouds acquired and collected in each first iteration process for each face before the current face.

[0147] Then, the viewpoint calculation module can calculate the viewpoint corresponding to each point cloud cluster in the plurality of point cloud clusters to obtain a plurality of viewpoints. Next, the viewpoint sorting module can sort the plurality of viewpoints in descending order of the density of the plurality of point cloud clusters to obtain a sorting result.

[0148] It should be noted that for each of the other faces, the second iteration module can calculate the point cloud corresponding to the start of the first iteration when the first iteration is first performed through the second iteration. And for each second iteration process, the second iteration module can perform the following operations until the second iteration terminates:

[0149] First, the viewpoint at the start of the current second iteration can be determined, and a point cloud can be acquired at the viewpoint, wherein the viewpoint at the start of the first second iteration can be the first viewpoint in the aforementioned sorting result.

[0150] Then, a first distance between the center of the point cloud collected in the second iteration and the center of the preset fused point cloud can be calculated, wherein the preset fused point cloud can refer to the fused point cloud obtained by the last first iteration for a face before the current face.

[0151] Next, in response to the first distance being greater than or equal to a preset threshold, the point cloud collected in the second iteration can be used as the point cloud corresponding to the start of the first iteration when the first iteration is performed for the first time for the current face.

[0152] Alternatively, in response to the first distance being less than the preset threshold, the next viewpoint adjacent to the viewpoint at the start of the second iteration can be selected from the sorting result as the viewpoint at the start of the next second iteration. In addition, the first distance can be, but is not limited to, a Euclidean distance.

[0153] According to the exemplary embodiments of the present disclosure, the aforementioned viewpoint calculation module can construct a reference coordinate system based on each point cloud cluster. The center of the point cloud cluster can be the origin of the reference coordinate system, and the normal vector of the point cloud cluster can be the z-axis of the reference coordinate system. The principal direction obtained by performing principal component analysis on the point cloud cluster can be the y-axis of the reference coordinate system, and the secondary direction obtained can be the x-axis of the reference coordinate system. Then, the viewpoint calculation module can calculate the viewpoint corresponding to the point cloud cluster based on the constructed reference coordinate system.

[0154] Figure 11 is a block diagram illustrating an electronic device 1100 according to an exemplary embodiment of the present disclosure.

[0155] Referring to Figure 11 The electronic device 1100 includes at least one memory 1101 having instructions stored therein and at least one processor 1102, which, when the instructions are executed, performs a three-dimensional measurement method of a strongly reflective object according to an exemplary embodiment of the present disclosure.

[0156] As an example, the electronic device 1100 can be a PC computer, a tablet device, a personal digital assistant, a smart phone, or other devices capable of executing the above instructions. Here, the electronic device 1100 is not necessarily a single electronic device, but can be a collection of any devices or circuits capable of executing the above instructions (or instruction sets) individually or jointly. The electronic device 1100 can also be part of an integrated control system or a system manager, or can be configured as a portable electronic device that interfaces with a local or remote device (e.g., via wireless transmission).

[0157] In electronic device 1100, processor 1102 can include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor can also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0158] Processor 1102 can execute instructions or code stored in memory 1101, which can also store data. The instructions and data can also be transmitted and received via a network through network interface device, which can employ any known transmission protocol.

[0159] Memory 1101 can be integrated with processor 1102, such as by being disposed within an integrated circuit microprocessor, etc. Further, memory 1101 can include a separate device, such as an external disk drive, a storage array, or other storage device usable by any database system. Memory 1101 and processor 1102 can be operatively coupled, or can communicate with each other, such as through I / O ports, network connections, etc., so that processor 1102 can read files stored in memory.

[0160] Further, electronic device 1100 can also include a video display, such as a liquid crystal display, and a user interface, such as a keyboard, a mouse, a touch input device, etc. All components of electronic device 1100 can be connected via a bus and / or network.

[0161] According to an example embodiment of the disclosure, a computer-readable storage medium can also be provided, which, when instructions stored therein are executed by a processor of an electronic device, enables the electronic device to perform the above-described three-dimensional measurement method of a strongly reflective object. Examples of the computer-readable storage medium herein include a read-only memory (ROM), a random access programmable read-only memory (PROM), an electrically erasable programmable read-only memory (EEPROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a non-volatile memory, a CD-ROM, a CD-R, a CD+R, a CD-RW, a CD+RW, a DVD-ROM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, a DVD-RAM, a BD-ROM, a BD-R, a BD-RLTH, a BD-RE, a Blu-ray or an optical disc memory, a hard disk drive (HDD), a solid state drive (SSD), a card memory such as a multimedia card, a secure digital (SD) card, or an extreme digital (XD) card, a magnetic tape, a floppy disk, a magneto-optical data storage device, an optical data storage device, a hard disk, a solid state disk, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-described computer-readable storage medium can be executed in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc., and in addition, in one example, the computer program and any associated data, data files, and data structures are distributed over a networked computer system so that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0162] According to an example embodiment of the disclosure, a computer program product can also be provided, which includes a computer program that, when executed by a processor, implements the three-dimensional measurement method of a strongly reflective object according to the disclosure.

[0163] According to the three-dimensional measurement method of a strongly reflective object and related devices of the disclosure, for each of the plurality of faces included in the object, the point cloud of the face can be calculated by controlling the robot to move multiple times in an iterative manner, i.e., the disclosure can control the robot to perform active motion to autonomously avoid the saturated area, i.e., the strong reflection area, on the object. In this way, accurate and complete point clouds can be ensured, and thus the three-dimensional information of the measured object can be ensured to conform to the actual situation of the object, thereby improving the accuracy of the three-dimensional measurement of the strongly reflective and textureless object.

[0164] According to exemplary embodiments of the present disclosure, robot strong reflection dodge translation motion will change the field of view of the camera and the illumination range of the projector at the current viewpoint, therefore, it is necessary to keep the consistency of the three-dimensional measurement area to the maximum by robot rotation motion.

[0165] Other embodiments of the present disclosure will be apparent to those of ordinary skill in the art from the disclosure and practice of the present disclosure. It is intended that the present disclosure cover any and all variations of the present disclosure that come within the scope of the four claims set forth below. The specification and examples are to be regarded as exemplary in nature and are not intended to limit the truly scope and spirit of the present disclosure.

[0166] It is to be understood that the present disclosure is not limited to the precise construction described above and shown in the attached drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the claims set forth below.

Claims

1. A three-dimensional measurement method for a highly reflective object, applied to a structured light system, characterized in that: The structured light system includes a robot, a projector, and a camera, wherein the projector and the camera are mounted at the end of a robotic arm of the robot, and the method includes: Obtain the point cloud corresponding to each of the multiple faces of the object; generating a three-dimensional point cloud of the object based on a point cloud corresponding to each of the plurality of surfaces; Wherein, for each of the surfaces, a point cloud corresponding to the surface is obtained through a first iteration; For each first iteration, perform the following operations until the first iteration is terminated: Obtaining a point cloud corresponding to the start of the first iteration, wherein, when the first iteration is initially performed, the point cloud corresponding to the start of the first iteration is determined based on a point cloud captured by the camera after the projector projects the fringe image onto the object; Calculate the translation vector based on the point cloud corresponding to the beginning of the first iteration; In response to the robot translating according to the translation vector, collecting a point cloud after the robot translates; Fusing the point cloud corresponding to the start of the first iteration with the point cloud collected after the robot has translated to obtain a fused point cloud; In response to the absence of a saturation boundary in the fused point cloud, using the fused point cloud as the point cloud corresponding to the surface; In response to the presence of a saturation boundary in the fused point cloud, the fused point cloud is used as the corresponding point cloud at the start of the next first iteration.

2. The three-dimensional measurement method according to claim 1, wherein: For the preset surface of the object, when the first iteration is initially performed, the corresponding point cloud at the beginning of the first iteration is the point cloud collected from the preset initial viewpoint.

3. The three-dimensional measurement method according to claim 2, wherein: The three-dimensional measurement method further includes: For each of the surfaces of the object other than the preset surface, density clustering is performed on the saturated area boundary point clouds in the plurality of point clouds to obtain a plurality of point cloud clusters, wherein the plurality of point clouds include: for each surface before the current surface, all point clouds acquired and collected in each first iteration process; Calculating a viewpoint corresponding to each point cloud cluster in the plurality of point cloud clusters to obtain a plurality of viewpoints; Sorting the plurality of viewpoints in descending order of density of the plurality of point cloud clusters to obtain a sorting result; For each of the other surfaces, calculating, by a second iteration, a point cloud corresponding to the start of the first iteration when the first iteration is initially performed; For each second iteration, the following operations are performed until the second iteration is terminated: Determine a viewpoint at the start of the second iteration, and collect a point cloud at the viewpoint, wherein the viewpoint at the start of the first second iteration is the first viewpoint in the sorting result; Calculating a first distance between the center of the point cloud collected in the second iteration and the center of a preset fused point cloud, where the preset fused point cloud is the fused point cloud obtained in the last first iteration for a face before the current face; In response to the first distance being greater than or equal to a preset threshold, using the point cloud collected in the second iteration as the point cloud corresponding to the start of the first iteration when the first iteration is initially performed on the current surface; In response to the first distance being less than the preset threshold, a next viewpoint adjacent to the viewpoint at the start of the current second iteration is selected from the sorting result as the viewpoint at the start of the next second iteration.

4. The three-dimensional measurement method according to claim 3, wherein: The calculating the viewpoint corresponding to each point cloud cluster in the plurality of point cloud clusters includes: Based on each point cloud cluster, a reference coordinate system is constructed, wherein the center of the point cloud cluster is the origin of the reference coordinate system, the normal vector of the point cloud cluster is the z-axis of the reference coordinate system, the main direction obtained by performing principal component analysis on the point cloud cluster is the y-axis of the reference coordinate system, and the secondary direction obtained is the x-axis of the reference coordinate system; Based on the reference coordinate system, the viewpoint corresponding to the point cloud cluster is calculated.

5. The three-dimensional measurement method according to claim 1, wherein: The three-dimensional measurement method further includes: For each first iteration, the following operations are also performed: determining a second distance between the center of the point cloud corresponding to the start of the first iteration and the plane of the projector; calculating a rotation angle based on the second distance; Perform a cross product of the translation vector and the principal axis vector of the projector to obtain a rotation axis; Calculating a rotation matrix based on the rotation axis and the rotation angle; In response to the robot rotating according to the rotation matrix, collecting a point cloud of the robot after the rotation; The step of fusing the point cloud corresponding to the start of the first iteration with the point cloud collected after the robot is translated includes: The point cloud corresponding to the beginning of the first iteration, the point cloud collected after the robot is translated, and the point cloud collected after the robot is rotated are fused to obtain the fused point cloud.

6. The three-dimensional measurement method according to claim 1, wherein: The calculation of the translation vector based on the point cloud corresponding to the start of the first iteration includes: Extract multiple saturation area boundary point clouds from the point cloud corresponding to the beginning of the first iteration; Calculating a three-dimensional oriented bounding box of each saturated region boundary point cloud in the plurality of saturated region boundary point clouds, and determining a vector of a shortest side among a plurality of sides included in the three-dimensional oriented bounding box; The translation vector is calculated based on vectors of multiple shortest sides of the multiple saturation area boundary point clouds in one-to-one correspondence.

7. A three-dimensional measurement device for highly reflective objects, applied to a structured light system, characterized in that: The structured light system includes a robot, a projector, and a camera. The projector and the camera are installed at the end of the robot's mechanical arm. The device includes: A single-face point cloud acquisition module is configured to acquire a point cloud corresponding to each of the multiple faces of the object; a three-dimensional point cloud generation module, configured to generate a three-dimensional point cloud of the object based on a point cloud corresponding to each of the multiple faces; Wherein, for each of the faces, the single-face point cloud acquisition module is configured to acquire the point cloud corresponding to the face through a first iteration; For each first iteration, perform the following operations until the first iteration is terminated: Obtaining a point cloud corresponding to the start of the first iteration, wherein, when the first iteration is initially performed, the point cloud corresponding to the start of the first iteration is determined based on a point cloud captured by the camera after the projector projects the fringe image onto the object; Calculate the translation vector based on the point cloud corresponding to the beginning of the first iteration; In response to the robot translating according to the translation vector, collecting a point cloud after the robot translates; Fusing the point cloud corresponding to the start of the first iteration with the point cloud collected after the robot has translated to obtain a fused point cloud; In response to the absence of a saturation boundary in the fused point cloud, using the fused point cloud as the point cloud corresponding to the surface; In response to the presence of a saturation boundary in the fused point cloud, the fused point cloud is used as the corresponding point cloud at the start of the next first iteration.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the three-dimensional measurement method of a highly reflective object according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the three-dimensional measurement method for a strongly reflective object according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the three-dimensional measurement method of a highly reflective object according to any one of claims 1 to 6 is implemented.

Images