An automated three-dimensional measurement method for multi-reflective surfaces based on irradiance fusion

Abstract

The present invention discloses an automated multi-reflective surface three-dimensional measurement method based on irradiance fusion. An exposure time selection network is used to generate a set of optimal exposure times and corresponding exposure times. There is no need to manually set the exposure time, which leads to redundancy in the captured images, thereby improving the automation and intelligence of the three-dimensional measurement process. A multiple irradiance fusion network is used to fuse the irradiance map sequences generated by all grayscale images to generate an irradiance map. On the one hand, this avoids directly using the grayscale map for phase calculation, and on the other hand, it reduces the error in the calculation using the irradiance map. At the same time, the multiple irradiance fusion network can feed back the irradiance map error value to the exposure time selection network for learning, thereby improving the accuracy of the phase calculation from the image perspective.

Description

Technical Field

[0001] The present invention belongs to the field of surface structured light three-dimensional measurement, and more specifically, relates to an automated multi-reflection surface three-dimensional measurement method based on irradiance fusion. Background Art

[0002] Accurately acquiring three-dimensional information about objects has long been a hot demand in the field of industrial measurement. Optical-based 3D measurement technology has been widely researched and applied due to its advantages, such as fast measurement speed, low hardware requirements, and high measurement accuracy. However, when measuring reflective objects, multiple reflections on the object surface can lead to dark or overexposed colors, resulting in uneven illumination in the fringe images captured by the camera. This results in an excessively large grayscale dynamic range, leading to loss of image features and hindering phase recovery and the calculation of the object's 3D information.

[0003] To address these issues, existing 3D reconstruction methods for objects with multiple reflective surfaces typically restore the camera's response function. During image capture, the photographer can adjust the image exposure by changing parameters such as the camera's aperture and exposure time. Multiple images with varying exposures are captured at the same location using different exposure methods. Because each brightness region exhibits different information, these images with varying exposure times are fused together to extract high-quality detail information from each image, resulting in a single image rich in surface information. However, insufficient exposure times during measurement can result in incomplete information in the fused image and large measurement errors. Excessive exposure times can lead to long image acquisition and post-processing times, while also placing high demands on device memory. Therefore, it's crucial to select an appropriate number of exposures and exposure times for each measurement object. However, for unknown measurement scenarios, it's often impossible to directly determine the required number of exposures and corresponding exposure times at the initial measurement stage. Relying on manual experience and lacking quantitative methods to select the appropriate number of exposures results in low measurement efficiency. Therefore, selecting an appropriate number of exposures and corresponding exposure time sequence is crucial to this technique. Summary of the Invention

[0004] In response to the above defects or improvement needs of the prior art, the present invention provides an automated multi-reflection surface three-dimensional measurement method based on irradiance fusion, the purpose of which is to determine a set of exposure times and exposure times of the initial image through an exposure time selection network, and the camera captures a set of image sequences at the corresponding exposure time, and restores the irradiance value of each set of images through the pixel value, and then inputs this set of irradiance values ​​into the multiple irradiance fusion network to obtain an accurate irradiance map, thereby reducing the influence of factors such as camera nonlinear response and multiple reflections on the object surface, and at the same time suppressing random noise in grayscale images, realizing high-precision phase calculation of grayscale images, and improving the smoothness of the three-dimensional reconstructed point cloud of tiny objects.

[0005] To achieve the above object, according to a first aspect of the present invention, there is provided an automated three-dimensional measurement method for multi-reflective surfaces based on irradiance fusion, comprising:

[0006] Training phase:

[0007] t1 and Input to the exposure time selection network, the left and right cameras collect a multiple exposure image sequence of sample j under the exposure time sequence output by the exposure time selection network, and input the irradiance map sequence of the multiple exposure image sequence into the multiple irradiance fusion network. The difference between the irradiance value output by the multiple irradiance fusion network and the irradiance true value is fed back to the exposure time selection network as the loss value, so as to jointly train the exposure time selection network and the multiple irradiance fusion network; wherein, the irradiance true value is the value of the irradiance at the preset multiple exposure time sequence t1, t2, ..., t i Multiple exposure image sequence acquired under The weighted fused irradiance value of the irradiance map sequence; is the exposure image of sample j at pose k obtained at the i-th exposure time;

[0008] Application stage:

[0009] S1: The projector projects a uniform white image, and the left and right cameras synchronously capture the projected image of the workpiece under test at a preset exposure time. The preset exposure time and the projected image at the preset exposure time are input into the trained exposure time selection network to predict the exposure time sequence;

[0010] S2, fixing the brightness of the projector and projecting a set of phase-shifted fringes onto the surface of the workpiece to be measured, and synchronously capturing a sequence of multiple exposure fringe images of the workpiece to be measured under the exposure time sequence predicted by the exposure time selection network with the left and right cameras, and inputting an irradiance map sequence of the multiple exposure fringe image sequence of the workpiece to be measured into a trained multiple irradiance fusion network, so that the trained multiple irradiance fusion network generates a fused irradiance map covering multi-reflection surface information of the workpiece to be measured;

[0011] S3, calculating the phase map of the fused irradiance map, obtaining an unfolded absolute phase map from the wrapped phase map, matching the phases of the left and right camera images along the epipolar lines, and calculating the three-dimensional point cloud of the workpiece to be measured based on the parallax of the same-name points in the left and right camera images.

[0012] Specifically, the phase is calculated using a fused irradiance map, and the unfolded absolute phase map is obtained from the wrapped phase map. The phases of the left and right camera images are matched along the epipolar direction, and then the three-dimensional point cloud of the object is calculated based on the parallax of the same-name points in the left and right camera images.

[0013] According to a second aspect of the present invention, there is provided an automated multi-reflective surface three-dimensional measurement system based on irradiance fusion, comprising: a computer-readable storage medium and a processor;

[0014] The computer-readable storage medium is used to store executable instructions;

[0015] The processor is configured to read the executable instructions stored in the computer-readable storage medium and execute the method according to the first aspect.

[0016] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0017] 1. Conventional camera exposure times for measuring multi-reflective objects are often manually set to capture multiple exposure images. This wide range of exposure times results in long acquisition times and significant projector temperature drift during measurement. The method provided by this invention uses an exposure time selection network to generate a set of optimal exposure times and corresponding exposure times. This eliminates the need for manual exposure time setting, which can lead to redundant captured images, and improves the automation and intelligence of the 3D measurement process.

[0018] 2. Traditional methods estimate irradiance maps by selecting or mixing pixels from images of different exposures, resulting in a large error range in the resulting irradiance histogram, making it difficult to remove images with large errors. The method provided by the present invention fuses the irradiance map sequences generated by all grayscale images into a single irradiance map through a multiple irradiance fusion network. This avoids directly using grayscale images for phase calculation and reduces the error associated with irradiance map calculations. Furthermore, the multiple irradiance fusion network can feed the irradiance map error value back to the exposure time selection network for learning, thereby improving the accuracy of phase calculations from an image perspective. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of the joint training process of the exposure time selection network and the multiple irradiance fusion network provided in an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of the structure of an exposure time selection network provided in an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of the structure of a multi-irradiance fusion network provided in an embodiment of the present invention;

[0022] Figure 4 A schematic flow chart of the application phase of the automated multi-reflective surface three-dimensional measurement method based on irradiance fusion provided in an embodiment of the present invention;

[0023] Figure 5 Schematic diagram of the camera imaging model of the 3D measurement system; DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0025] Most existing methods for selecting the number of exposures either seek a minimum bounding set that covers the dynamic range of the entire measurement scene or maximize the image's signal-to-noise ratio. While these methods offer good performance, they generally directly calculate phase using grayscale values. As can be seen from the image acquisition process, grayscale calculation depends on the light intensity, exposure time, and camera response function used during camera acquisition. Therefore, they can often only process images that are linearly dependent on the scene irradiance. Unlike pixel grayscale values, the irradiance reflected from an object's surface is inherently determined by illumination and reflectivity, unconstrained by the camera's dynamic response range and unrelated to saturation. Furthermore, like pixel values, irradiance values ​​change synchronously with the phase shift process, making them suitable for phase recovery. However, current irradiance calculations, determined by the number of exposures and the corresponding exposure time, can result in significant errors in irradiance calculations for areas with uneven brightness on multi-reflective surfaces, thus affecting the accuracy of phase calculations.

[0026] Based on this, an embodiment of the present invention provides an automated multi-reflective surface three-dimensional measurement method based on irradiance fusion, which is used to adaptively select a set of optimal exposure time sequences. By fusing multiple irradiance maps, the error in irradiance value calculation is reduced, thereby improving the accuracy of phase calculation and ensuring the accuracy and applicability of three-dimensional measurement of multi-reflective objects. The method includes:

[0027] Training phase:

[0028] like Figure 1 As shown, t1 and Input to the exposure time selection network, the left and right cameras collect a multiple exposure image sequence of sample j under the exposure time sequence output by the exposure time selection network, and input the irradiance map sequence of the multiple exposure image sequence into the multiple irradiance fusion network. The difference between the irradiance value output by the multiple irradiance fusion network and the irradiance true value is fed back to the exposure time selection network as the loss value, so as to jointly train the exposure time selection network and the multiple irradiance fusion network; wherein, the irradiance true value is the value of the irradiance at the preset multiple exposure time sequence t1, t2, ..., t i Multiple exposure image sequence acquired under The weighted fused irradiance value of the irradiance map sequence; is the exposure image of sample j at posture k obtained at the i-th exposure time; i = 1, 2, …, I, j = 1, 2, …, J, k = 1, 2, …, K, I, J, K are all positive integers greater than 1.

[0029] Specifically, a convolutional neural network for predicting multiple exposure time series (i.e., exposure time selection network) and a convolutional neural network for predicting fused irradiance maps (i.e., multiple irradiance fusion network) are constructed.

[0030] Construct a large-scale dataset of multiple exposures of multi-reflective surface objects, keep the projection intensity unchanged, and obtain the multiple exposure projection images of the sample according to a fixed multiple exposure time sequence (i.e., a preset multiple exposure time sequence), including t1, t2, ..., t i and And the irradiance value generated by weighted fusion of the corresponding irradiance map.

[0031] Using the above dataset, the exposure time selection network and the multiple irradiance fusion network are jointly iteratively trained: the multiple irradiance fusion network feeds back the error between the generated irradiance data and the true irradiance value to the exposure time selection network, and the joint training is continuously performed to finally obtain the trained exposure time selection network and the multiple irradiance fusion network:

[0032] Further, if Figure 2 As shown, the multiple irradiance fusion network is a void convolutional neural network.

[0033] The multi-irradiance fusion network is defined as follows:

[0034] y H =HDRNet(Y s ) (1)

[0035] Among them, y H Irradiance map generated by the multi-irradiance fusion network, Y s It is a sequence of irradiance images at different exposure times calculated according to the camera response function.

[0036] Further, if Figure 3 As shown, the exposure time selection network is a convolutional neural network.

[0037] The exposure time selection network is defined as: inputting the preview image X of the camera with a fixed exposure time, the network predicts and outputs a set of k different exposure time sequences Y = {y0,y1,...,y K-1} to cover the dynamic range of the entire scene as much as possible.

[0038] In the training phase, samples t1 and t2 are randomly selected from the training set. And set t1 and is the initial state. Figure 1 As shown in the figure, the exposure time selection network model takes the initial state value as the input of the convolutional neural network. Through the combination of convolutional layers, the fully connected layer outputs a set of appropriate exposure time sequences; the multiple irradiance fusion network takes a set of calculated irradiance maps as input, and through the operation of the convolutional layer, outputs a fused irradiance map.

[0039] The multiple irradiance fusion network is a dilated convolutional neural network, and its network structure is as follows: Figure 2 As shown in the figure, the irradiance of multiple sets of exposure images is used as input; after inputting into the neural network, the exposure images are convolved in sequence, and then the output results of each convolutional layer are finally connected together and convolved again, and finally the predicted irradiance map is output.

[0040] Furthermore, the irradiance value output by the multiple irradiance fusion network is V total , the true value of irradiance corresponding to the training data set is V true ,According to the error of irradiance value, the exposure time selection network state is updated, specifically:

[0041] t new =t m +ΔV loss *s (2)

[0042] Where, ΔV loss is the loss of irradiance, ΔV loss =V total -V true , t m is the initial exposure time of the mth training, s is the coefficient factor, t new Exposure time after being updated based on irradiance loss feedback.

[0043] During the training process, the parameter optimization of the exposure time selection network is determined by the image irradiance value fed back by the multiple irradiance fusion network. At the same time, the learning and fusion of the multiple irradiance fusion network are affected by the number of exposures and exposure time generated by the exposure time selection network. Therefore, it is necessary to jointly train the two networks using a multiple exposure dataset. The joint training process of the exposure time selection network and the multiple irradiance fusion network includes:

[0044] (1) Randomly select samples t1 and t2 from the training set And set t1 and is the initial state;

[0045] (2) Based on the current state, the convolutional layer of the exposure time selection network combines the exposure image with the initial time according to the input to extract features, and discretizes a set of increasing exposure time sequences through the fully connected layer;

[0046] (3) The left and right cameras collect images in the time sequence to calculate the camera response function, and the irradiance map sequence under this exposure time sequence is calculated based on the camera response function;

[0047] (4) The irradiance map sequence of the exposure image is input into the multi-irradiance fusion network, and its convolutional layer will fuse them into a single irradiance map based on the clarity of the local features of the image;

[0048] (5) Calculate the loss of the multi-irradiance fusion network based on the current irradiance value and the true irradiance value, feed it back to the exposure time selection network, update the current state, repeat steps (2)-(4) until the current state meets the set conditions, and continue training the next set of samples;

[0049] (6) Determine whether the network parameter update setting is satisfied. If so, perform the network update and return to step (2) to start a new round until the preset number of training steps is reached and training is stopped. If not, do not update and return to step (2) to start a new round until the preset number of training steps is reached and training is stopped. The network parameter update setting is the preset number of training steps. The network model at this time meets the generalized irradiance calculation requirements of most related workpieces.

[0050] The multiple exposure projection image sequence is obtained synchronously by the left and right cameras, that is, a uniform white image is projected onto the surface of the sample by the projector, and the left and right cameras synchronously obtain the multiple exposure projection image sequence of the sample in a preset multiple exposure time sequence; the preset multiple exposure time sequence and the multiple exposure projection image sequence obtained by the left camera and / or the right camera are used as the data set. For example, when J = 5 and K = 13, the multiple exposure projection image sequences in the preset multiple exposure time sequence t1, t2, ..., t i In the following example, the exposure images of 5 samples in 13 positions (i.e., 5*13*i images are acquired by the left camera and the right camera respectively) are used as the data set.

[0051] Application stage, such as Figure 4 As shown:

[0052] S1: The projector projects a uniform white image, and the left and right cameras synchronously capture the projection image of the workpiece to be measured at a preset exposure time. The preset exposure time and the projection image at the preset exposure time are input into the trained exposure time selection network to predict the exposure time sequence.

[0053] Specifically, the projector projects a uniform white image, and the left and right cameras synchronously capture the projection image under the camera's fixed low exposure time t1; the above-mentioned low exposure time and projection image are passed into the trained exposure time selection network to obtain the number of exposure times and the corresponding exposure time sequence.

[0054] S2: The brightness of the fixed projector remains unchanged, and the projector projects a set of phase-shifted stripes onto the surface of the workpiece to be measured. The left and right cameras synchronously collect a multiple-exposure stripe image sequence of the workpiece to be measured under the exposure time sequence predicted by the exposure time selection network. The irradiance map sequence calculated from the multiple-exposure stripe image sequence of the workpiece to be measured is input into the trained multiple irradiance fusion network. The trained multiple irradiance fusion network generates a fused irradiance map covering the multi-reflection surface information of the workpiece to be measured.

[0055] Furthermore, an irradiance map sequence of the multiple exposure projection image sequence is calculated according to a response function of the camera.

[0056] Furthermore, the phase-shift stripes are three-frequency four-step phase-shift stripes.

[0057] Specifically, the projector brightness is fixed and constant, and phase-shifted fringes are projected onto the surface of the workpiece to be measured. The left and right cameras synchronously capture a sequence of multiple exposure fringe images of the workpiece to be measured under the exposure time sequence. Based on the camera response function, an irradiance map sequence of the multiple exposure fringe image sequence of the workpiece to be measured is calculated and input into a trained multiple irradiance fusion network to obtain a fused irradiance map. Specifically, the exposure time sequence predicted by the trained exposure time selection network is projected onto the object to be measured, and the cameras synchronously capture multiple exposure fringe images. Irradiance maps at different exposure times are calculated based on the camera response function. This set of irradiance maps is then fed into the trained multiple exposure fusion network to fuse the irradiance map sequence to obtain a single irradiance map.

[0058] S3, calculating the phase map of the fused irradiance map, obtaining an unfolded absolute phase map from the wrapped phase map, matching the phases of the left and right camera images along the epipolar lines, and calculating the three-dimensional point cloud of the workpiece to be measured based on the parallax of the same-name points in the left and right camera images.

[0059] Specifically, the phase map is calculated based on the above-mentioned irradiance map, and the wrapped phase map is expanded into an absolute phase map through the multi-frequency heterodyne principle. The phases of the left and right camera images are matched along the epipolar lines, and the three-dimensional point cloud of the measured object is calculated based on the parallax of the same-name points in the left and right camera images.

[0060] Since the irradiance value of the multi-reflection on the surface of an object is a physical quantity, it can be expressed as a sine wave in the following formula (3):

[0061]

[0062] Among them, E a Indicates the average brightness, E m represents the modulation depth, φ(x,y) represents the phase information, Indicates the ideal reference phase value. Irradiance value E n Like the pixel grayscale value, it changes synchronously with the phase shift process, so it can be used for phase recovery.

[0063] Therefore, it is necessary to estimate the camera response function of the stripe image and restore the irradiance value through the pixel grayscale value. In most cases, the camera response function is a nonlinear function. The relationship between the image irradiance and the scene radiance is as follows:

[0064]

[0065] Among them, E n is the image irradiance, L is the scene radiance, f o is the focal length, s is the aperture size, is the angle between the light and the optical axis. For the imaging system, the irradiance value calculation formula is shown in formula (5):

[0066] z nj =f(E n t j ) (5)

[0067] Where f is the irradiance response curve, t j is the exposure time of the jth image. Then the inverse of the response curve can be expressed as:

[0068] f -1 (z nj )=E n t j (6)

[0069] Based on formula (6), the camera response function curve can be reconstructed, and then the irradiance value can be restored by the inverse function of the camera response function, as shown in formula (7):

[0070]

[0071] Where g(.)=f -1 is the inverse function of the camera response function, ω is the weight function, I nj is the pixel grayscale value, and P is the number of images collected with different exposure times.

[0072] Furthermore, the method for calculating the phase from the irradiance map output by the multiple irradiance fusion network is shown in formula (8):

[0073]

[0074] Then, the multi-frequency phase recovery method is used to obtain the unfolded absolute phase map from the wrapped phase map, and the three-dimensional point cloud is reconstructed by stereo matching of the left and right images.

[0075] like Figure 5 As shown, the measurement system used in the present invention is a surface structured light three-dimensional measurement system, including left and right cameras and a projector, and the models and parameters of the left and right cameras are consistent. It can be understood that there will be certain errors in the internal parameters of the camera during calibration.

[0076] An embodiment of the present invention provides an automated multi-reflective surface three-dimensional measurement system based on irradiance fusion, comprising: a computer-readable storage medium and a processor;

[0077] The computer-readable storage medium is used to store executable instructions;

[0078] The processor is configured to read the executable instructions stored in the computer-readable storage medium and execute the method described in any one of the above embodiments.

[0079] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An automated three-dimensional measurement method for multi-reflective surfaces based on irradiance fusion, characterized in that: include: Training phase: t1 and Input to the exposure time selection network, the left and right cameras collect a multiple exposure image sequence of sample j under the exposure time sequence output by the exposure time selection network, and input its irradiance map sequence into the multiple irradiance fusion network, and the difference between the irradiance value output and the irradiance true value is fed back to the exposure time selection network as the loss value, so as to jointly train the exposure time selection network and the multiple irradiance fusion network; wherein, the irradiance true value is the value of the irradiance in the preset multiple exposure time sequence t1, t2, ..., t i Multiple exposure image sequence acquired under The weighted fused irradiance value of the irradiance map sequence; is the exposure image of sample j at pose k obtained at the i-th exposure time; Application stage: S1: The projector projects a uniform white image, and the left and right cameras synchronously capture the projected image of the workpiece under test at a preset exposure time. The preset exposure time and the projected image at the preset exposure time are input into the trained exposure time selection network to predict the exposure time sequence; S2, fixing the brightness of the projector and projecting phase-shifted fringes onto the surface of the workpiece to be measured, and synchronously collecting a sequence of multiple exposure fringes of the workpiece to be measured under the exposure time sequence predicted by the exposure time selection network with the left and right cameras, and inputting the irradiance map sequence into the trained multiple irradiance fusion network to obtain a fused irradiance map; S3, calculating the phase map of the fused irradiance map, obtaining an unfolded absolute phase map from the wrapped phase map, matching the phases of the left and right camera images along the epipolar lines, and calculating the three-dimensional point cloud of the workpiece to be measured based on the parallax of the same-name points in the left and right camera images.

2. The method according to claim 1, wherein The multiple irradiance fusion network is a dilated convolutional neural network.

3. The method according to claim 1 or 2, wherein: The exposure time selection network is a convolutional neural network.

4. The method according to claim 1, wherein An irradiance map sequence of the multiple exposure image sequence is calculated according to a response function of the camera.

5. The method according to claim 1 or 2, wherein: The phase-shift stripes are three-frequency four-step phase-shift stripes.

6. The method according to claim 1, wherein The measurement is performed using a surface structured light 3D measurement system, including left and right cameras and a projector, and the left and right cameras have the same type parameters.

7. An automated multi-reflective surface three-dimensional measurement system based on irradiance fusion, characterized in that: include: Computer-readable storage media and processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read the executable instructions stored in the computer-readable storage medium and execute the method according to any one of claims 1 to 6.

Images