An Adaptive Projection Three-Dimensional Measurement Method and System Based on Virtual Image Rendering

Abstract

The present invention discloses an adaptive projection three-dimensional measurement method and system based on virtual image rendering, which establishes an image rendering model that couples projection imaging characteristics with scale-adaptive voxel multi-dimensional features, predicts the camera reflectivity based on a convolutional neural network, and then calculates the optimal projection intensity according to the camera reflectivity map. This new attempt is expected to get rid of the dependence on actual images and achieve automatic viewpoint planning in a simulation environment. Among them, a data-driven neural network is used for learning and training, and the optimal projection intensity is calculated according to different reflection situations, greatly improving the robustness of the measurement. The method provided by the present invention automatically calculates adaptive projection parameters according to the reflection characteristics of the current voxel surface, obtains high-quality three-dimensional data with the fewest viewpoints, can overcome the problem of difficult selection of projection intensity parameters in traditional methods, and provides the necessary basic data and evaluation parameters for viewpoint planning.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional measurement of surface structured light, and more specifically, relates to an adaptive projection three-dimensional measurement method and system based on virtual image rendering. Background Art

[0002] Currently, in the field of industrial part measurement, non-contact measurement technologies such as line lasers and surface structured light are combined with motion mechanisms such as industrial robots to achieve automated and rapid three-dimensional measurement of complex parts, which has been widely applied. However, this technology still has the problem of difficult automatic optimization of measurement view point configuration parameters. Surfaces of complex parts such as aero-engine blades and casings usually exhibit multiple reflection characteristics (specular reflection, weak reflection, multiple reflections, etc.), which affect the quality of measurement data. Therefore, when planning measurement view points, it is necessary to synchronously set the exposure time sequence of this view point. However, at present, the interaction mechanism between the surface characteristics of the object and the projected surface structured light is not clear, and it is difficult to render a realistic measurement image in the simulation environment, resulting in the inability to synchronously and automatically optimize the measurement configuration parameters when generating view points. Summary of the Invention

[0003] In view of the above defects or improvement requirements of the prior art, the present invention provides an adaptive projection three-dimensional measurement method and system based on virtual image rendering, thereby solving the technical problem that it is difficult for the prior art to balance the quality and efficiency of measurement.

[0004] To achieve the above object, according to the first aspect of the present invention, there is provided an adaptive projection three-dimensional measurement method based on virtual image rendering, including:

[0005] S1, creating a three-dimensional voxel grid of the workpiece to be measured, storing the light source position and camera position information of the workpiece to be measured, converting it to the camera coordinate system and then inputting it into an image rendering prediction model to predict the camera reflectivity map of the workpiece to be measured at the light source position and camera position;

[0006] Wherein, the image rendering prediction model includes a 3D convolutional network, a projection unit and a 2D convolutional network connected in sequence; after the three-dimensional voxel grid input into the image rendering prediction model is subjected to three-dimensional convolution by the 3D convolutional network, it is projected from the three-dimensional space to the two-dimensional space by the projection unit to obtain a two-dimensional voxel grid; the 2D convolutional network predicts and outputs the camera reflectivity map of the workpiece to be measured at the light source position and camera position;

[0007] S2, obtaining an adaptive stripe projection image of the projector according to the camera reflectivity map, the preset optimal intensity of each pixel in the two-dimensional voxel grid, and the correspondence between each pixel in the two-dimensional voxel grid in the camera plane and the projector plane.

[0008] S3. Project the adaptive fringe projection image onto the workpiece to be measured. The left and right cameras acquire images of the workpiece to be measured at this time, calculate the phase map and perform matching, and reconstruct the three-dimensional point cloud of the workpiece to be measured.

[0009] According to the second aspect of the present invention, there is provided an adaptive projection three-dimensional measurement device based on virtual image rendering, including:

[0010] An input module for creating a three-dimensional voxel grid of the workpiece to be measured, storing the light source position and camera position information of the workpiece to be measured, converting it to the camera coordinate system and inputting it into an image rendering prediction model to predict the camera reflectivity map of the workpiece to be measured at the light source position and camera position;

[0011] Wherein, the image rendering prediction model includes a 3D convolutional network, a projection unit and a 2D convolutional network connected in sequence; after the three-dimensional voxel grid input into the image rendering prediction model is subjected to three-dimensional convolution by the 3D convolutional network, it is projected from the three-dimensional space to the two-dimensional space by the projection unit to obtain a two-dimensional voxel grid; the 2D convolutional network predicts and outputs the camera reflectivity map of the workpiece to be measured at the light source position and camera position;

[0012] An adaptive fringe projection image acquisition module for calculating the optimal projection intensity of each pixel in the two-dimensional voxel grid according to the camera reflectivity map; and acquiring the adaptive fringe projection image of the projector according to the optimal projection intensity and the corresponding relationship between each pixel in the camera plane and the projector plane;

[0013] A three-dimensional point cloud reconstruction module for projecting the adaptive fringe projection image onto the workpiece to be measured. The left and right cameras acquire images of the workpiece to be measured at this time, calculate the phase map and perform matching, and reconstruct the three-dimensional point cloud of the workpiece to be measured.

[0014] According to the third aspect of the present invention, there is provided an adaptive projection three-dimensional measurement system based on virtual image rendering, including: a computer-readable storage medium and a processor;

[0015] The computer-readable storage medium is used to store executable instructions;

[0016] The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the method described in the first aspect.

[0017] Generally speaking, compared with the prior art through the above technical solutions conceived by the present invention, the following beneficial effects can be achieved:

[0018] 1. Compared with existing methods that all need to analyze the optimal projection intensity through pre-shot camera images, which makes it difficult to achieve automatic planning, the method provided by the present invention establishes an image rendering model that couples projection imaging characteristics with multi-dimensional features of scale-adaptive voxels, predicts the camera reflectivity based on a convolutional neural network, and then calculates the optimal projection intensity according to the camera reflectivity map. This new attempt is expected to get rid of the dependence on actual images and achieve automatic viewpoint planning in a simulation environment.

[0019] 2. The method provided by the present invention takes into account that the reflection characteristics of the part surface will affect the quality of measurement data, uses a data-driven neural network for learning and training, and calculates the best projection intensity according to different reflection situations, greatly improving the robustness of the measurement.

[0020] 3. The method provided by the present invention automatically calculates adaptive projection parameters according to the reflection characteristics of the current voxel surface, which is a necessary condition for obtaining high-quality three-dimensional data with the fewest viewpoints and is also a basic problem for realizing high-quality measurement planning; the method provided by the present invention can overcome the problem of difficult selection of projection intensity parameters in traditional methods and provide the necessary basic data and evaluation parameters for viewpoint planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic structural diagram of the image rendering prediction model provided by an embodiment of the present invention;

[0022] Figure 2 It is a view of the adaptive projection process provided by an embodiment of the present invention;

[0023] Figure 3 It is a schematic diagram of the projector mask generation provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0025] Many automated generation methods for measuring configuration parameters have been proposed in the prior art. Existing methods for measuring multi-reflection surfaces can be mainly divided into two categories: single-shot optimal measurement and multi-measurement fusion. Multi-measurement fusion is to fuse the images obtained under multiple different camera exposure conditions to enhance the dynamic imaging range. Due to the advantages of simple principle and high measurement accuracy, this method has begun to be widely used. However, at the present stage, the setting of the exposure time sequence still depends on manual experience. And this method highly relies on the camera images of the measurement viewpoints pre-shot by the measurement system, resulting in difficulty in directly automatically planning the measurement viewpoints and their exposure time sequence parameters in the simulation environment.

[0026] In contrast, single-shot optimal measurement is to calculate the optimal measurement configuration parameters at the current measurement viewpoint through methods such as adaptive projection. The adaptive projection intensity refers to the method of matching the image obtained by the camera with the projector image to adjust the brightness of the local area or even a single pixel to enhance the imaging dynamic range. This idea can specifically change the projection intensity of each pixel to achieve pixel-level intensity adjustment. At the same time, the parameters of both the camera and the projector can be set in the simulation environment, and the images captured by the camera can also be predicted through a learning framework. Therefore, the single-measurement method can obtain good measurement results in the simulation environment.

[0027] Based on this, the present invention provides an adaptive projection three-dimensional measurement method based on virtual image rendering. This method can automatically and accurately generate the optimal projection intensity parameters according to the given measurement viewpoints in the simulation environment, improving the quality and efficiency of measurement. The method includes:

[0028] S1, creating a three-dimensional voxel grid of the workpiece to be measured, storing the light source position and camera position information of the workpiece to be measured, converting it to the camera coordinate system and then inputting it into the image rendering prediction model to predict the camera reflectivity map of the workpiece to be measured at the light source position and camera position.

[0029] Furthermore, the three-dimensional voxel grid of the workpiece to be measured also stores the specular reflection coefficient and material texture features of its surface.

[0030] Specifically, create a three-dimensional voxel grid of the workpiece to be measured, encode the specular reflection coefficient, material texture features of the object surface, and light source position into a multi-dimensional vector and store it in the corresponding grid information to obtain a multi-dimensional tensor; then apply a rigid body transformation to convert it from the world coordinate system to the camera coordinate system.

[0031] When performing coordinate transformation, embed the input voxel grid into a larger grid to ensure that the object will not be cut off after rotation. Then, perform parametric transformation by rotating around the y-axis (pitch angle) and z-axis (yaw angle), and the distance R (scaling factor, that is, the distance between the object and the camera).

[0032] Further, the camera position information includes visibility features.

[0033] The light source position information includes the three-dimensional coordinates and orientation (represented by a normal vector) of the light source.

[0034] Among them, as Figure 1 shown, the image rendering prediction model includes a 3D convolutional network, a projection unit, and a 2D convolutional network connected in sequence; after the three-dimensional voxel grid input to the image rendering prediction model is subjected to three-dimensional convolution by the 3D convolutional network, it is projected from the three-dimensional space to the two-dimensional space by the projection unit to obtain a two-dimensional voxel grid; the 2D convolutional network predicts and outputs the camera reflectivity map of the workpiece to be measured under the light source position and the camera position.

[0035] Specifically, the 3D convolutional network is used to perform three-dimensional convolution on the input, and the projection unit is used to project the grid from the three-dimensional space to the two-dimensional space to obtain a two-dimensional voxel grid; the 2D convolutional network is used to predict and output the camera reflectivity map of the workpiece to be measured under the light source position and the camera position.

[0036] Further, the camera position information can also be stored in the grid information after the three-dimensional voxel grid input to the image rendering prediction model is subjected to three-dimensional convolution by the 3D convolutional network.

[0037] First, perform three-dimensional convolution on the input three-dimensional voxel grid converted to the camera coordinate system, and then calculate the visibility features of the object (i.e., the workpiece to be measured) in the scene according to the camera position and store them in the voxel grid information. It can be understood that the reflectivity coefficient and material texture features are all stored in the feature channels of the voxel grid by the visibility features.

[0038] After that, design a projection unit to fuse the depth and feature layer (i.e., the feature channel) dimensions of the voxel grid, so as to project the grid from the three-dimensional space to the two-dimensional space.

[0039] Specifically, use feature mapping to fold the depth dimension and map the input four-dimensional tensor to a three-dimensional compressed tensor V' with dimensions of W×H×(D·C), where W, H, D, and C represent width, height, depth, and feature channel respectively.

[0040] The projection unit can be expressed as:

[0041] I i,j,k =f(∑ω k,dc ·V' i,j,dc +b k )

[0042] where \(i\) and \(j\) are pixel coordinates, \(k\) is the image channel, \(dc\) is the compressed depth channel, where \(d\) and \(c\) are the depth and channel dimensions of the original four-dimensional tensor respectively, and \(f\) is some non-linear function.

[0043] Finally, a 2D convolutional network is constructed. Using the camera reflectivity map generated from the multiple-exposure images of the object to be measured actually obtained as data-driven, the 2D convolutional network is trained, and the prediction of the camera reflectivity map of the 2D grid image under different illuminations and camera positions can be achieved (i.e., predicting the camera reflectivity map of the 2D voxel grid output by the projection unit under different light source positions and camera positions).

[0044] Among them, the light source and camera positions are set according to actual measurement requirements.

[0045] Furthermore, the 2D convolutional network takes as input the three-dimensional voxel grid in which the light source position and camera position information on the surface of the sample are stored and converted to the camera coordinate system, and is obtained after training with the camera reflectivity map generated from the multiple-exposure images of the sample actually obtained as the ground truth.

[0046] Furthermore, when the three-dimensional voxel grid of the workpiece to be measured also stores the specular reflection coefficient and material texture characteristics on the surface of the workpiece to be measured, the three-dimensional voxel grid of the sample also stores the specular reflection coefficient and material texture characteristics on the surface of the sample.

[0047] S2. According to the camera reflectivity map, the preset optimal intensity of each pixel in the two-dimensional voxel grid, and the correspondence between each pixel in the two-dimensional voxel grid in the camera plane and the projector plane, an adaptive fringe projection image of the projector is obtained.

[0048] Specifically, as Figure 2 shown, according to the camera reflectivity map and the preset optimal intensity of each pixel in the two-dimensional voxel grid, the optimal projection intensity of each pixel (i.e., the optimal projection intensity of the projector) is obtained; according to the optimal projection intensity of the projector and the correspondence between each pixel in the two-dimensional voxel grid in the camera plane and the projector plane, the optimal projection intensity of the projector is converted to the projector coordinate system. Finally, an adaptive fringe projection image of the projector is obtained according to the optimal projection intensity of the projector after coordinate transformation.

[0049] The intensity of each pixel in the two-dimensional voxel grid can be expressed as:

[0050]

[0051] where is the intensity of each pixel, that is, the pixel intensity captured by the camera, \(k\) is the camera sensitivity, \(t\) is the exposure time. is the reflectance estimate for each pixel r(x,y) (obtained from the camera reflectance map), L p (x,y) is the projected light intensity, is the estimated value of the intensity of the ambient light and the mutual reflection of the surface

[0052] For simplicity, the parameters k and t, are both set to constants. Therefore, when the camera obtains the best intensity, the projector light intensity can be expressed as:

[0053]

[0054] where L opt (x,y) is the best projection intensity for each pixel in the two-dimensional voxel grid (i.e., the best projection intensity of the projector), I ideal is the preset best intensity for each pixel in the two-dimensional voxel grid, that is, the ideal capture intensity of the camera. This value can be set according to the actual situation. For example, take I ideal = 220.

[0055] According to the camera reflectance map and the preset best intensity of each pixel in the two-dimensional voxel grid, obtain the best projection intensity L opt (x,y) of the projector. After that, since L opt (x,y) is under the camera imaging plane (x,y), it is necessary to convert L opt (x,y) to the projector imaging plane (u,v) according to the correspondence relationship of each pixel in the two-dimensional voxel grid between the camera plane and the projector plane.

[0056] As Figure 3 shown, the coordinate mapping relationship between the camera imaging plane and the projector imaging plane can be expressed as:

[0057]

[0058] where H is a 3×3 homography matrix and s is a scalar factor.

[0059] L opt (x,y) after being mapped to the projector imaging plane (u,v), the intensity modulation I”(u,v) of the fringe image can be expressed as:

[0060]

[0061] where L min (u,v) is the minimum intensity of the fringe image.

[0062] The average intensity I'(u,v) can be expressed as:

[0063]

[0064] The adaptive fringe image for three-dimensional measurement projection can be generated through I”(u, v) and I'(u, v). The intensity calculation of the adaptive fringe image can be expressed as:

[0065]

[0066] where α is the phase of the fringe image and n represents different image sequences.

[0067] Furthermore, the method for obtaining the correspondence of each pixel between the camera plane and the projector plane is as follows:

[0068] Project a series of fringe images onto the workpiece to be measured. The left and right cameras acquire the images of the workpiece to be measured at this time, solve the absolute phase map, and map its coordinates to the imaging plane of the projector to establish the coordinate mapping relationship of each pixel in the two-dimensional voxel grid between the camera plane and the projector plane.

[0069] Among them, the camera imaging model is:

[0070] I(x, y) = A(x, y) + B(x, y)cos(φ(x, y) + θ)

[0071] In the formula, A(x, y) = α·t(βI a + I 0 ) + μ is the average gray-scale distribution of the image without grating projection; (x, y) are the coordinates; B(x, y) = α·t(βI p ) is the gray-scale modulation of the image, that is, the influence of the projected grating image on the gray scale of the camera image; φ(x, y) is the relative phase value; θ is the phase shift.

[0072] The phase calculation often adopts the three-frequency four-step method, where three frequencies are used to unwrap the phase and four steps refer to the four-step phase-shifting method.

[0073] The basic principle of the four-step phase-shifting method is to first obtain four vertical grating images with different phase shifts, and then calculate the relative phase values of the corresponding points on the images according to the camera imaging model. The phase shifts of the four grating images are 0, π, and the gray-scale distributions of their camera images are as follows:

[0074]

[0075] Then the relative phase value can be calculated by the following formula:

[0076]

[0077] Based on the multi-frequency heterodyne principle, the relative phase value is unwrapped to obtain the corresponding absolute phase value.

[0078] S3. Project the adaptive fringe projection image onto the workpiece to be measured. The left and right cameras acquire the images of the workpiece to be measured at this time, calculate the phase map and perform matching, and reconstruct the three-dimensional point cloud of the workpiece to be measured.

[0079] Specifically, in the actual scenario, an adaptive fringe image is projected and generated. The left and right cameras acquire the images of the object to be measured at this time, calculate the phase map and perform matching, and reconstruct the three-dimensional point cloud.

[0080] The measurement system adopted by the present invention is a structured light three-dimensional measurement system for a surface, including a left camera, a right camera and a projector, and the parameters of the left and right cameras are the same.

[0081] The method provided by the present invention establishes an image rendering convolutional neural network model that couples a projection imaging relationship and a voxel coding feature matrix, can realize the prediction of the camera reflectivity map, and then calculates the optimal projection intensity according to the camera reflectivity map. This new attempt is expected to get rid of the dependence on actual images and realize automatic viewpoint planning in a simulation environment; the method proposed by the present invention provides a new adaptive digital fringe projection technology, avoids image saturation, and has a high signal-to-noise ratio and a wide range of reflectivity changes in three-dimensional shape measurement.

[0082] An embodiment of the present invention provides an adaptive projection three-dimensional measurement device based on virtual image rendering, including:

[0083] An input module, configured to create a three-dimensional voxel grid of the workpiece to be measured, store the light source position and camera position information of the workpiece to be measured, convert it to the camera coordinate system, and then input it into the image rendering prediction model to predict the camera reflectivity map of the workpiece to be measured at the light source position and camera position;

[0084] Wherein, the image rendering prediction model includes a 3D convolutional network, a projection unit and a 2D convolutional network connected in sequence; after the three-dimensional voxel grid input into the image rendering prediction model is subjected to three-dimensional convolution by the 3D convolutional network, it is projected from the three-dimensional space to the two-dimensional space by the projection unit to obtain a two-dimensional voxel grid; the 2D convolutional network predicts and outputs the camera reflectivity map of the workpiece to be measured at the light source position and camera position;

[0085] An adaptive fringe projection image acquisition module, configured to calculate the optimal projection intensity of each pixel in the two-dimensional voxel grid according to the camera reflectivity map; and acquire the adaptive fringe projection image of the projector according to the optimal projection intensity and the corresponding relationship between each pixel in the camera plane and the projector plane;

[0086] A three-dimensional point cloud reconstruction module is used to project the adaptive fringe projection image onto the workpiece to be measured. The left and right cameras acquire images of the workpiece to be measured at this time, calculate the phase map and perform matching, and reconstruct the three-dimensional point cloud of the workpiece to be measured.

[0087] An embodiment of the present invention provides an adaptive projection three-dimensional measurement system based on virtual image rendering, including:

[0088] A computer-readable storage medium and a processor;

[0089] The computer-readable storage medium is used to store executable instructions;

[0090] The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the method described in the first aspect.

[0091] Those skilled in the art can easily understand that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An adaptive projection three-dimensional measurement method based on virtual image rendering, characterized in that, Including: S1. Create a three-dimensional voxel grid of the workpiece to be measured, store the light source position and camera position information of the workpiece to be measured, convert it to the camera coordinate system and then input it into the image rendering prediction model to predict the camera reflectivity map of the workpiece to be measured at the light source position and camera position; Wherein, the image rendering prediction model includes a 3D convolutional network, a projection unit and a 2D convolutional network connected in sequence; the three-dimensional voxel grid input into the image rendering prediction model is subjected to three-dimensional convolution by the 3D convolutional network and then projected from the three-dimensional space to the two-dimensional space by the projection unit to obtain a two-dimensional voxel grid; the 2D convolutional network predicts and outputs the camera reflectivity map of the workpiece to be measured at the light source position and camera position; S2. According to the camera reflectivity map, the preset optimal intensity of each pixel in the two-dimensional voxel grid, and the corresponding relationship of each pixel in the two-dimensional voxel grid between the camera plane and the projector plane, obtain the adaptive fringe projection image of the projector; S3. Project the adaptive fringe projection image onto the workpiece to be measured, and the left and right cameras obtain the image of the workpiece to be measured at this time, calculate the phase map and perform matching to reconstruct the three-dimensional point cloud of the workpiece to be measured.

2. The method according to claim 1, characterized in that, The 2D convolutional network is obtained by training with the three-dimensional voxel grid storing the light source position and camera position information on the surface of the sample and converted to the camera coordinate system as the input, and using the camera reflectivity map generated from the multiple exposure images of the actually obtained sample as the ground truth.

3. The method according to claim 2, characterized in that, The three-dimensional voxel grids of the workpiece to be measured and the sample also store the specular reflection coefficient and material texture features on their surfaces.

4. The method according to claim 1, characterized in that, The camera position information includes visibility features.

5. The method according to claim 1, characterized in that, The method for obtaining the corresponding relationship of each pixel between the camera plane and the projector plane is as follows: Project a series of fringe images onto the workpiece to be measured, the left and right cameras obtain the image of the workpiece to be measured at this time, solve the absolute phase map, and map its coordinates to the projector imaging plane to obtain the corresponding relationship of each pixel between the camera plane and the projector plane.

6. An adaptive projection three-dimensional measurement device based on virtual image rendering, characterized in that, Including: An input module, configured to create a three-dimensional voxel grid of the workpiece to be measured, store the light source position and camera position information of the workpiece to be measured, convert it to the camera coordinate system and then input it into the image rendering prediction model to predict the camera reflectivity map of the workpiece to be measured at the light source position and camera position; Wherein, the image rendering prediction model includes a 3D convolutional network, a projection unit and a 2D convolutional network connected in sequence; the three-dimensional voxel grid input into the image rendering prediction model is subjected to three-dimensional convolution by the 3D convolutional network and then projected from the three-dimensional space to the two-dimensional space by the projection unit to obtain a two-dimensional voxel grid; the 2D convolutional network predicts and outputs the camera reflectivity map of the workpiece to be measured at the light source position and camera position; An adaptive fringe projection image acquisition module, configured to calculate the optimal projection intensity of each pixel in the two-dimensional voxel grid according to the camera reflectivity map; according to the optimal projection intensity and the corresponding relationship of each pixel between the camera plane and the projector plane, obtain the adaptive fringe projection image of the projector; A three-dimensional point cloud reconstruction module, which is used to project the adaptive fringe projection image onto the workpiece to be measured. The left and right cameras acquire images of the workpiece to be measured at this time, calculate the phase map and perform matching, and reconstruct the three-dimensional point cloud of the workpiece to be measured.

7. An adaptive projection three-dimensional measurement system based on virtual image rendering, characterized in that, It includes: A computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the method according to any one of claims 1-5.

Images