Method for predicting exposure duration of structured light camera based on RGB information

Abstract

The invention discloses a structured light camera exposure duration prediction method based on RGB information. The method is divided into a camera calibration stage and a camera working stage. In a camera calibration stage, firstly, a two-dimensional color image calibration plate is shot with different exposure durations to obtain a corresponding two-dimensional color image sequence and a three-dimensional point cloud image sequence, and the two-dimensional color image sequence and the three-dimensional point cloud image sequence are used to calibrate a camera response function of RGB three channels of a camera and cut-off exposure of the RGB three-color channels corresponding to complete point cloud information. In the working stage of the camera, a two-dimensional color image sequence of a working scene is firstly shot, and the optimal exposure duration of the three-dimensional point cloud image of the structured light camera is calculated by using the camera data obtained in the calibration stage. The method provided by the invention has good adaptability to complex ambient light, and compared with manual adjustment of exposure duration, the three-dimensional point cloud imaging speed of the structured light camera adopting the method provided by the invention is greatly increased, and the imaging quality is obviously improved.

Description

Technical Field

[0001] The present invention relates to a method for predicting the exposure duration of a structured light camera in the field of three-dimensional measurement, and more particularly to a method for predicting the exposure duration of a structured light camera based on RGB information. Background Art

[0002] A structured light camera usually consists of a projector and an image sensor as the acquisition hardware. First, the camera controller controls the projector to emit a coded pattern. After being reflected by an object within the working scene range, the image sensor captures the reflected pattern. By using a decoding algorithm in the camera controller to decode the two-dimensional fringe image captured by the image sensor, the corresponding spatial point coordinate positions are calculated, and the point cloud data information of the object surface is generated. Therefore, the premise for a structured light camera to calculate depth information is that its image sensor can capture a clear two-dimensional fringe reflection image for calculation. However, when the ambient light is too strong, the brightness of the object surface far exceeds the dynamic range of the camera image sensor itself, resulting in overexposure of both the fringe area pattern and the non-fringe area pattern of the two-dimensional pattern captured by the image sensor. The camera cannot obtain a clearly contrasted fringe reflection pattern, and the local loss of the coded pattern information causes a decrease in measurement accuracy or even inability to calculate the point information, resulting in point cloud holes.

[0003] Currently, the general solution is to manually adjust the exposure duration based on experience. At the same time, in the research field, to solve the problem of missing images in three-dimensional point cloud photography, researchers generally achieve this from three aspects: hardware improvement, image fusion, or fringe pattern intensity adjustment. In terms of hardware improvement, researchers add auxiliary devices to the structured light camera so that the camera can obtain a complete two-dimensional fringe pattern for three-dimensional information calculation, but this increases the complexity of the camera structure. In terms of image fusion or fringe pattern intensity adjustment, researchers control the intensity of different regions of the projector separately or fuse the two-dimensional fringe images to obtain a complete two-dimensional fringe pattern for three-dimensional information calculation, but most of them are still in the research stage and the technology is not yet mature. Currently, the mainstream structured light cameras on the market still do not have the function of adaptive exposure. In the actual engineering application field, structured light camera manufacturers usually do not open advanced interfaces such as adjusting projection patterns and customizing sine fringes for camera users to conduct research and development. In current actual production, in the face of complex ambient light, technicians still need to manually adjust based on experience.

[0004] Therefore, for the users of structured light cameras and aiming at the existing ordinary structured light cameras on the market, a method needs to be proposed. This method can predict the appropriate exposure duration for three-dimensional point cloud by using the two-dimensional color image captured by the structured light camera, and capture images with the appropriate exposure duration, so as to generate the optimal two-dimensional image and three-dimensional point cloud image, thereby reducing the burden on users. Summary of the Invention

[0005] To solve the problems and requirements in the background technology, the present invention proposes a method for predicting the exposure duration of a structured light camera based on RGB information.

[0006] The technical solution of the present invention is as follows:

[0007] Step 1: Build a two-dimensional color image calibration board;

[0008] Step 2: The structured light camera sequentially captures the two-dimensional color image calibration board according to the exposure durations in the exposure duration calibration sequence, obtaining a sequence of two-dimensional color images of the calibration board and a sequence of three-dimensional point cloud images of the calibration board;

[0009] Step 3: Perform calibration based on the exposure duration calibration sequence, the sequence of two-dimensional color images of the calibration board, and the sequence of three-dimensional point cloud images of the calibration board to obtain the camera response function and the cut-off exposure amounts of the RGB three color channels that can correctly capture effective three-dimensional point cloud information;

[0010] Step 4: The structured light camera sequentially captures the working scene according to the exposure durations in the working exposure duration sequence, obtaining a sequence of two-dimensional scene color images;

[0011] Step 5: Use the camera response function sequence to convert the sequence of two-dimensional scene color images into a true irradiance image, and then calculate the maximum exposure duration of the three-dimensional point cloud image corresponding to each pixel in the true irradiance image according to the cut-off exposure amounts of the RGB three color channels that can correctly capture effective three-dimensional point cloud information, thereby obtaining an exposure duration matrix; finally, determine the optimal exposure duration of the three-dimensional point cloud image of the structured light camera according to the exposure duration matrix.

[0012] In the said Step 1, the two-dimensional color image calibration board is composed of seven juxtaposed gradient color bands of the same length and width, and the pixel values (R, G, B) of the RGB three color channels in the seven gradient color bands are specifically as follows:

[0013] Gradually changing from (255, 255, 0) to (255, 255, 255), from (255, 0, 255) to (255, 255, 255), from (0, 255, 255) to (255, 255, 255), from (255, 0, 0) to (255, 255, 255), from (0, 255, 0) to (255, 255, 255), from (0, 0, 255) to (255, 255, 255), and from (0, 0, 0) to (255, 255, 255).

[0014] In the said Step 2, the minimum value T of the exposure durations in the exposure duration calibration sequence min is the shortest exposure duration required for the two-dimensional color pictures of the structured light camera and the exposure duration of the shortest three-dimensional point cloud image The maximum value in max is the longest exposure duration of the two-dimensional color image required by the structured light camera and the longest exposure duration of the three-dimensional point cloud image The minimum value in

[0015] Specifically, step 3 is as follows:

[0016] Step 3.1: According to the exposure duration calibration sequence and the corresponding calibration plate two-dimensional color image sequence, calibrate to obtain the camera response function sequences of the R, G, and B color channels;

[0017] Step 3.2: Calculate the RGB three-color channel thresholds at which the structured light camera can correctly capture effective three-dimensional point cloud information according to the calibration plate two-dimensional color image sequence and the calibration plate three-dimensional point cloud image sequence;

[0018] Step 3.3: Use the camera response function sequences of the RGB three color channels to convert the RGB three-color channel thresholds into the cut-off exposure amounts of the RGB three color channels that can correctly capture effective three-dimensional point cloud information.

[0019] Specifically, step 3.2 is as follows:

[0020] First, generate the calibration plate three-dimensional point cloud image sequence P 3D The corresponding point cloud height binary mask matrix sequence P 3D_P , where the mask corresponding to the missing part of the point cloud information is set to 1, and the mask corresponding to the complete part of the point cloud information is set to 0; then, each two-dimensional calibration plate color image in the calibration plate two-dimensional color image sequence P 2D is corresponding to the binary mask matrix in the binary mask matrix sequence P 3D_P at the same exposure duration. The RGB three-color information of the calibration plate color image corresponding to the position where the value in the mask matrix is 0 is set to 0 to obtain a new two-dimensional calibration plate color image. Process the images and matrices in the two sequences in turn to obtain a new calibration plate two-dimensional color image sequence NP 2D ; Calculate the RGB three-color channel pixel value thresholds at which effective three-dimensional point cloud information can be correctly captured according to the new calibration plate two-dimensional color image sequence NP 2D

[0021] In step 3.2, calculating the RGB three-color channel pixel value thresholds at which effective three-dimensional point cloud information can be correctly captured according to the new calibration plate two-dimensional color image sequence NP 2D is specifically as follows:

[0022] First, from the new calibration plate two-dimensional color image sequence NP 2D ​The effective pixel points of all two-dimensional calibration plate color images form an effective pixel set A point ; Then count the effective pixel set A point The pixel value combinations of all RGB three channels in and the number of their occurrences After that, remove the effective pixels corresponding to the pixel value combinations whose number is less than the preset effective pixel quantity threshold, and obtain a new effective pixel set NA point ; Then calculate the equivalent pixel value H point corresponding to each effective pixel in the new effective pixel set NA RGB , and use the R value, G value, and B value of the effective pixel with the smallest equivalent pixel value H point in the new effective pixel set NA RGB as the pixel value thresholds Z of the RGB three color channels that can correctly capture the effective three-dimensional point cloud information r , Z g , Z b .

[0023] The calculation formula of the equivalent pixel value H RGB corresponding to each effective pixel is as follows:

[0024]

[0025] where H RGB is the equivalent pixel value corresponding to the pixel, is the R channel pixel value of the pixel point, is the G channel pixel value of the pixel point, is the B channel pixel value of the pixel point.

[0026] In step 5, using the camera response function sequence to convert the two-dimensional scene color image sequence into a true irradiance image includes:

[0027] For each two-dimensional scene color image in the two-dimensional scene color image sequence WP, use the camera response functions of the R, G, and B channels respectively to convert the pixel values of the corresponding color channels of each two-dimensional scene color image into exposure values, and then divide the exposure values by the corresponding exposure duration to obtain the two-dimensional scene relative irradiance image corresponding to the two-dimensional scene color image. After traversing and processing all the two-dimensional scene color images in the two-dimensional scene color image sequence WP, finally obtain a two-dimensional scene relative irradiance image sequence EP; perform weighted processing on the irradiance map values at the same pixel positions in the two-dimensional scene relative irradiance image sequence EP according to the following formula to form a true two-dimensional scene absolute irradiance image E:

[0028]

[0029] where EP xrepresents the relative irradiance value of the pixel in the x-th two-dimensional scene relative irradiance map, N is the number of pictures in the two-dimensional scene relative irradiance map sequence EP, and WP x (γ) represents the pixel value of channel γ in the x-th two-dimensional scene color map in the two-dimensional scene color map sequence WP, where γ is one of the channels R, G, and B; V is the weight function.

[0030] In step 5, calculating the maximum exposure duration of the three-dimensional point cloud image corresponding to each pixel in the true irradiance image according to the cut-off exposure amounts of the RGB three color channels that can correctly capture effective three-dimensional point cloud information includes:

[0031] The maximum exposure duration T of the three-dimensional point cloud image corresponding to the pixel (i, j) in the exposure duration matrix ij(min) satisfies the following formula:

[0032] T ij(min) = min(T ij(r) , T ij(g) , T ij(b) )

[0033]

[0034] where, E ij(r) , E ij(g) and E ij(b) respectively refer to the absolute irradiance of the R channel, the absolute irradiance of the G channel, and the absolute irradiance of the B channel of the pixel (i, j) in the true irradiance image; T ij(r) , T ij(g) and T ij(b) respectively refer to the maximum exposure durations corresponding to the R, G, and B channels of the pixel (i, j); X r , X g and X b are respectively the cut-off exposure amounts of the R, G, and B three color channels that can correctly capture effective three-dimensional point cloud information.

[0035] In step 5, determining the optimal exposure duration of the three-dimensional point cloud image of the structured light camera according to the exposure duration matrix, specifically:

[0036] First, divide the duration values in the exposure duration matrix into intervals and count the number of pixels falling into each interval, and then perform clustering processing on the intervals according to the number of pixels in each interval; after completing the clustering, select the lower limit value of the interval with the smallest value in each cluster as the candidate optimal exposure duration; according to the sorting of the candidate exposure durations, select the exposure duration in turn according to the support quantity of the camera.

[0037] Compared with the prior art, the beneficial effects of the present invention are:

[0038] (1) The present invention provides a method for predicting the exposure duration of structured light for a structured light camera without an adaptive exposure function. After calibration, only a few two-dimensional color images need to be captured by the structured light camera to predict several appropriate exposure durations for the depth map. The complete three-dimensional point cloud information can be conveniently obtained by controlling the exposure duration of the structured light camera. This method has good adaptability to complex ambient light. Compared with manually adjusting the exposure duration, the imaging speed of the structured light camera using the method proposed by the present invention is significantly accelerated, and the imaging quality is significantly improved.

[0039] (2) Predicting the exposure duration at the absolute irradiance level can effectively avoid interference from camera parameters such as the camera aperture and exposure duration. Only by capturing two-dimensional color images to obtain the absolute irradiance information of the real scene, there is no need to capture again, saving the imaging duration.

[0040] (3) By introducing the method of exposure duration clustering, the exposure duration distribution corresponding to all pixels can be obtained, which is convenient for quickly selecting appropriate exposure durations according to different requirements in the future. Description of the Drawings

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. These exemplary descriptions do not constitute a limitation on the embodiments. Unless otherwise stated, the figures in the drawings do not constitute a proportional limitation. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0042] Figure 1 It is a flowchart of a method for predicting the exposure duration of a structured light camera based on RGB information provided by the present invention.

[0043] Figure 2 It is a schematic diagram of a calibration board in an embodiment of the present invention.

[0044] Figure 3 It is a schematic diagram of the three-channel camera response function in an embodiment of the present invention.

[0045] Figure 4 It is a schematic diagram of the absolute irradiance in an embodiment of the present invention.

[0046] Figure 5This is a comparison diagram of the Z-axis perspective image in the three-dimensional point cloud information of the calibration board in the embodiment of the present invention with the corresponding binary image and the two-dimensional color image of the calibration board. Among them, (a) is the Z-axis perspective view of the three-dimensional point cloud information of the calibration board taken at a certain exposure duration. The black area is the overexposed area of the point cloud, and other colors are the schematic Z values in the Z-axis direction of the three-dimensional point cloud information; (b) is a schematic binary mask image generated by setting the pixel values of the missing three-dimensional information part in the three-dimensional point cloud image to 1 and the pixels corresponding to the normal three-dimensional information to 0; (c) is a new two-dimensional color image generated after processing according to the schematic binary mask image.

[0047] Figure 6 This is a comparison diagram of the three-dimensional point cloud image in the embodiment of the present invention with the two-dimensional color image processed by the RGB three-channel threshold. Among them, (a) is the Z-axis perspective view of the three-dimensional point cloud randomly selected for the exposure duration. The black area is the overexposed area, and other colors are the schematic Z values in the Z-axis direction of the three-dimensional point cloud; (b) is the two-dimensional image at the same exposure duration, and the image obtained by resetting the RGB values of all pixels exceeding the threshold in the image to 0.

[0048] Figure 7 This is the exposure duration statistical histogram in the embodiment of the present invention. Detailed implementation manners

[0049] In order to enable those skilled in the art of this technology to better understand the technical solutions in this application, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only specific embodiments of the present invention, rather than all embodiments. Usually, the components of the embodiments of the present invention described and shown in the accompanying drawings here can be arranged and designed in various different configurations. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present invention.

[0050] Reference Figure 1, the method of the present invention is divided into two stages. The first is the structured light camera calibration stage, and the second is the structured light camera working stage. In the structured light camera calibration stage, while keeping other parameters of the structured light camera unchanged, the structured light camera is controlled to photograph a calibration board by setting several different exposure duration parameters. At each exposure duration, two images of the calibration board need to be taken, namely a two-dimensional color image and a three-dimensional point cloud image. The camera response function can be calibrated using the sequence of two-dimensional color images taken under different exposure duration parameters based on the relationship between the real environment exposure value and the pixel value. By calibration using the sequence of two-dimensional color images and three-dimensional point cloud images taken under different exposure duration parameters, the pixel value threshold corresponding to the minimum natural light intensity that enables the structured light camera to sufficiently distinguish the stripe patterns projected by the projector can be obtained. Due to different shooting characteristics of the camera, there will be differences in the pixel values of the natural light intensities of the red, green, and blue channels in the captured images. Therefore, it is necessary to calibrate different thresholds for the red, green, and blue, i.e., RGB channels. Combining the already calibrated camera response function, the respective exposure amounts corresponding to the thresholds of the RGB channels can be calculated. In the structured light camera working stage, first, the structured light camera is controlled to photograph the workpiece by setting several different exposure duration parameters. At each exposure duration, only the two-dimensional color image of the workpiece needs to be taken. Since projecting patterns by the structured light camera is not required for taking the two-dimensional color image, the exposure time can be greatly reduced. Using the sequence of the captured two-dimensional color images and based on the camera response function in the calibration process, the irradiance image of the actual scene can be obtained. According to the relationship between irradiance and exposure amount, the optimal exposure duration corresponding to each pixel can be solved. By statistically analyzing the optimal exposure durations of all pixels and making a histogram, the global optimal exposure duration can be obtained.

[0051] The specific method proposed by the present invention includes the following steps:

[0052] Step 1: In the calibration stage of the structured light camera, build a two-dimensional color image calibration board;

[0053] As Figure 2 shown, the two-dimensional color image calibration board consists of seven longitudinally juxtaposed gradient color bands of the same length and width. The seven gradient color bands serve as marker patterns, and the color information in the color bands serves as marker points. The pixel values (R, G, B) of the RGB three color channels in the seven gradient color bands in the RGB color gamut are specifically as follows:

[0054] Gradually change from (255, 255, 0) to (255, 255, 255), from (255, 0, 255) to (255, 255, 255), from (0, 255, 255) to (255, 255, 255), from (255, 0, 0) to (255, 255, 255), from (0, 255, 0) to (255, 255, 255), from (0, 0, 255) to (255, 255, 255), and from (0, 0, 0) to (255, 255, 255).

[0055] Step 2: The structured light camera sequentially captures the two-dimensional color image calibration board according to the exposure duration in the exposure duration calibration sequence (i.e., using the exposure duration in the exposure duration calibration sequence as the exposure duration parameter of the structured light camera), and obtains a sequence of calibration board two-dimensional color images and a sequence of calibration board three-dimensional point cloud images;

[0056] In Step 2, the minimum value T of the exposure duration in the exposure duration calibration sequence min is the shortest exposure duration required for the two-dimensional color images of the structured light camera and the shortest exposure duration of the three-dimensional point cloud images The maximum value T of the exposure duration in the exposure duration calibration sequence max is the longest exposure duration required for the two-dimensional color images of the structured light camera and the longest exposure duration of the three-dimensional point cloud images The interval between two consecutive exposure durations is a fixed duration Δt. The exposure duration value in the exposure duration calibration sequence T is t 1 , t 2 , ···, t i , ···, t N , N is the number of sequence elements, t i is the i-th exposure duration, t i = T min +(i - 1)·Δt, t i ≤ T max , represents the set of non-negative integers.

[0057] Keeping other parameters of the structured light camera unchanged, at each exposure duration t i simultaneously control the camera exposure parameters to capture the two-dimensional color image P of the two-dimensional color image calibration board i 2D and the three-dimensional point cloud image P i 3D . After sequentially capturing according to the exposure duration in the exposure duration calibration sequence, a sequence of calibration board two-dimensional color images P 2D and a sequence of calibration board three-dimensional point cloud images P 3D, satisfy

[0058] Step 3: Calibrate to obtain a sequence of camera response functions and the cut-off exposure amounts of the RGB three color channels that can correctly capture effective three-dimensional point cloud information according to the exposure duration calibration sequence, the two-dimensional color image sequence of the calibration board, and the three-dimensional point cloud image sequence of the calibration board;

[0059] Specifically, Step 3 is as follows:

[0060] Step 3.1: According to the exposure duration calibration sequence and the corresponding two-dimensional color image sequence of the calibration board, use the Debevec algorithm to calibrate and obtain the sequence of camera response functions corresponding to the R, G, and B three color channels. The formula is as follows:

[0061] f r (x r ) = z r

[0062] f g (x g ) = z g

[0063] f b (x b ) = z b

[0064] Among them, f r (), f g (), f b () are the camera response functions of the R, G, and B three color channels respectively, representing the conversion relationships between the pixel values z r , z g , z b of the three color channels and their corresponding exposure amount values x r , x g , x b , as shown in Figure 3 ;

[0065] Step 3.2: Calculate the thresholds of the RGB three color channels that the structured light camera can recognize and correctly capture effective three-dimensional point cloud information according to the two-dimensional color image sequence of the calibration board and the three-dimensional point cloud image sequence of the calibration board. Specifically:

[0066] First, generate a sequence of point cloud height binary mask matrices P 3D corresponding to the three-dimensional point cloud image sequence of the calibration board, where the masks corresponding to the missing parts of the point cloud information are set to 1, and the masks corresponding to the complete parts of the point cloud information are set to 0. Each binary mask matrix only contains 1 and 0; then, each two-dimensional calibrated board color image in the two-dimensional color image sequence P 3D_P of the calibration board is combined with the sequence of binary mask matrices P 2D of the calibration board.3D_P It corresponds to the binary mask matrix under the same exposure duration in [description], and sets the RGB three-color information of the calibration plate color image corresponding to the positions with a value of 0 in the mask matrix to 0, obtaining a new two-dimensional calibration plate color image. Process the images and matrices in the two sequences in turn to obtain a new sequence of two-dimensional calibration plate color images NP 2D , That is According to the new sequence of two-dimensional calibration plate color images NP 2D Calculate the pixel value thresholds Z r , Z g , Z b 。

[0067] In step 3.2, use the following formula to generate a sequence of calibration plate three-dimensional point cloud images P 3D The point cloud height binary mask matrix corresponding to each three-dimensional calibration plate point cloud image in [description], and its element values are:

[0068]

[0069] Where z represents the Z-direction height value in the three-dimensional information of a certain pixel in the three-dimensional calibration plate point cloud image, α represents the value at the corresponding position in the binary mask matrix, and K represents the effective range of the Z-axis in the three-dimensional point cloud information supported by the structured light camera.

[0070] In step 3.2, according to the new sequence of two-dimensional calibration plate color images NP 2D Calculate the pixel value thresholds of the RGB three-color channels, specifically:

[0071] First, all valid pixel points (i.e., points where all three RGB channels are non-zero) of the two-dimensional calibration plate color images in the new sequence of two-dimensional calibration plate color images NP 2D form a valid pixel set A point ; then count all the pixel value combinations point of the three RGB channels in the valid pixel set A and the number of times they appear Then remove the valid pixels corresponding to the pixel value combinations with a number less than the preset valid pixel quantity threshold (set to 10 in this embodiment) to obtain a new valid pixel set NA point , and the set NA point contains all the pixel value combinations of the three RGB channels as Then calculate the equivalent pixel value H point corresponding to each valid pixel in the new valid pixel set NA RGB , and for the equivalent pixel value H point in the new valid pixel set NA RGBThe R value, G value, and B value of the smallest valid pixel are used as the RGB three-color channel pixel value threshold Z for correctly capturing valid three-dimensional point cloud information r , Z g , Z b .

[0072] The equivalent pixel value H corresponding to each valid pixel RGB is calculated as follows:

[0073]

[0074] where H RGB is the equivalent pixel value corresponding to the pixel, is the R-channel pixel value of the pixel point, is the G-channel pixel value of the pixel point, is the B-channel pixel value of the pixel point.

[0075] In this embodiment, the Z-axis perspective view of the calibration board three-dimensional point cloud information captured under a certain exposure duration is as shown in Figure 5 (a). First, a binary matrix is created from the three-dimensional point cloud map. In the binary matrix, the part corresponding to the missing three-dimensional information in the three-dimensional point cloud is set to 1, and the pixels corresponding to the normal three-dimensional information are set to 0. The binary image is as shown in Figure 3 (b); the binary image and the two-dimensional color image under the same exposure duration are processed. If the corresponding binary image is 0 at the pixel position of the two-dimensional color image, the RGB information in the two-dimensional color image at that pixel position is all set to 0, otherwise it remains unchanged. The new two-dimensional color image generated in this way only contains the ambient light RGB values in the case of overexposure of the three-dimensional point cloud, as shown in Figure 5 (c). All images are processed in turn. Statistical analysis is performed on the non-zero pixels and their RGB values in all the new two-dimensional color images to remove abnormal situations. After squaring each of the three values of each remaining color point and adding them together, the color point with the minimum result is selected, and its corresponding RGB values are respectively used as the maximum thresholds for the three channels. After testing, the threshold has a relatively high credibility, as shown in Figure 5 (a) is a three-dimensional point cloud map with a randomly selected exposure duration. The black area is the overexposed area, and the other colors are the schematic diagrams of the Z values in the Z-axis direction of the three-dimensional point cloud map. Figure 6 (b) is the two-dimensional color image under the same exposure duration, and the image obtained by resetting the RGB values of the pixel points whose RGB channels all exceed the threshold to 0. Figure 6 (a) is a three-dimensional point cloud map with a randomly selected exposure duration. The black area is the overexposed area, and the other colors are the schematic diagrams of the Z values in the Z-axis direction of the three-dimensional point cloud map. Figure 6 (b) is the two-dimensional color image under the same exposure duration, and the image obtained by resetting the RGB values of the pixel points whose RGB channels all exceed the threshold to 0.

[0076] Step 3.3: Based on the RGB three-color channel pixel value threshold, calculate the cut-off exposure amount values X of the RGB three-color channels that can correctly capture valid three-dimensional point cloud information through the camera response function obtained in Step 3.1 r , Xg , X b 。

[0077] Step 4: During the working stage of the structured light camera, the structured light camera sequentially captures the working scene according to the exposure durations in the working exposure duration sequence to obtain a sequence of two-dimensional scene color images;

[0078] Step 5: Use the sequence of camera response functions to convert the sequence of two-dimensional scene color images into real irradiance images, as Figure 4 shown. Then, according to the cut-off exposure amounts of the RGB three color channels that can correctly capture effective three-dimensional point cloud information generated in Step 3, calculate the maximum exposure duration of the three-dimensional point cloud image corresponding to each pixel in the real irradiance image, so as to obtain an exposure duration matrix; finally, determine the optimal exposure duration of the three-dimensional point cloud image of the structured light camera according to the exposure duration matrix for the structured light camera to capture the optimal three-dimensional point cloud image.

[0079] In Step 5, using the sequence of camera response functions to convert the sequence of two-dimensional scene color images into real irradiance images includes:

[0080] For each two-dimensional scene color image in the sequence WP of two-dimensional scene color images, use the camera response functions of the R, G, and B channels respectively to convert the pixel values of the corresponding color channels of each two-dimensional scene color image into exposure amount values, and then divide the exposure amount values by the corresponding exposure durations to obtain the two-dimensional scene relative irradiance map corresponding to the two-dimensional scene color image. After traversing and processing all the two-dimensional scene color images in the sequence WP of two-dimensional scene color images, finally obtain a sequence EP of two-dimensional scene relative irradiance maps, and the sequence contains N images in total; perform weighted processing on the irradiance map values at the same pixel positions in the sequence EP of two-dimensional scene relative irradiance maps according to the following formula to form the real two-dimensional scene absolute irradiance map E:

[0081]

[0082] where EP x represents the relative irradiance value of the pixel in the x-th two-dimensional scene relative irradiance map, N is the number of pictures in the sequence EP of two-dimensional scene relative irradiance maps, WP x (γ) represents the pixel value of channel γ in the x-th two-dimensional scene color image in the sequence WP of two-dimensional scene color images, and γ is one of R, G, and B; V is the weight function.

[0083] Substitute and calculate the pixel values of the R, G, and B channels of the two-dimensional scene color images and the relative irradiance values in the corresponding two-dimensional scene relative irradiance maps at the same exposure duration in turn.

[0084] In step 5, calculate the maximum exposure duration of the 3D point cloud image corresponding to each pixel in the true irradiance image according to the cut-off exposure amounts of the RGB three color channels that can correctly capture the effective 3D point cloud information generated in step 3, including:

[0085] The following relationship is satisfied between the absolute irradiance and the exposure amount:

[0086] X = E·Δt;

[0087] where X refers to the image exposure amount, E refers to the absolute irradiance, and Δt refers to the exposure duration.

[0088] The maximum exposure duration T of the 3D point cloud image corresponding to the pixel (i, j) in the exposure duration matrix ij(min) satisfies the following formula:

[0089] T ij(min) = min(T ij(r) , T ij(g) , T ij(b) )

[0090]

[0091] where E ij(r) , E ij(g) and E ij(b) respectively refer to the absolute irradiance of the R channel, the absolute irradiance of the G channel, and the absolute irradiance of the B channel of the pixel (i, j) in the true irradiance image; T ij(r) , T ij(g) and T ij(b) respectively refer to the maximum exposure durations corresponding to the R, G, and B channels of the pixel (i, j); X r , X g and X b are respectively the cut-off exposure amount values of the R, G, and B three color channels that can correctly capture the effective 3D point cloud information.

[0092] In step 5, determine the optimal exposure duration of the 3D point cloud image of the structured light camera according to the exposure duration matrix, specifically:

[0093] First, divide the interval of all duration values in the exposure duration matrix and count the number of pixels falling into each interval, and then perform clustering processing on the intervals according to the number of pixels in each interval; after completing the clustering, select the lower limit value of the interval with the smallest value from each cluster as the candidate optimal exposure duration; according to the sorting of the candidate exposure durations, select the exposure duration in ascending order. If the camera only supports one exposure duration, select the smallest one among the candidate exposure durations; if the camera supports multiple exposure durations, select several of the smallest candidate exposure durations according to the supported quantity, or select the appropriate exposure duration according to the requirements.

[0094] In this embodiment, two three-dimensional exposure duration clustering results in bright and dark scenes can be obtained through clustering. The minimum values of each cluster are T1 and T2 respectively. To take into account both the brightness of the bright and dark environments and reduce the point cloud noise, the T1 duration is used to capture the bright area to prevent overexposure of the point cloud, and the T2 duration is used to capture the dark area to reduce the influence of point cloud noise.

[0095] Figure 7 The exposure duration distribution and aggregation in the exposure duration matrix are shown in the form of a histogram.

[0096] Finally, it should be noted that the above embodiments and descriptions are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced. Without departing from the spirit and scope of the disclosure of the technical solutions of the present invention, they should all be covered by the protection scope of the claims of the present invention.

Claims

1. A method for predicting exposure time of a structured light camera based on RGB information, characterized in that: The steps include: Step 1: Build a 2D color image calibration plate; Step 2: The structured light camera sequentially photographs the two-dimensional color image calibration plate according to the exposure time in the exposure time calibration sequence to obtain a two-dimensional color image sequence of the calibration plate and a three-dimensional point cloud image sequence of the calibration plate; Step 3: Calibrate according to the exposure time calibration sequence, the calibration plate 2D color image sequence and the calibration plate 3D point cloud image sequence to obtain the camera response function and the cutoff exposure of the RGB three color channels that can correctly capture the effective 3D point cloud information; Step 4: The structured light camera sequentially photographs the working scene according to the exposure time in the working exposure time sequence to obtain a two-dimensional scene color image sequence; Step 5: Use the camera response function sequence to convert the two-dimensional scene color image sequence into a real irradiance image, and then calculate the maximum exposure time of the three-dimensional point cloud image corresponding to each pixel in the real irradiance image according to the cutoff exposure of the RGB three color channels that can correctly capture the effective three-dimensional point cloud information, so as to obtain the exposure time matrix; finally, determine the optimal exposure time of the three-dimensional point cloud image of the structured light camera according to the exposure time matrix.

2. According to the method for predicting exposure time of a structured light camera based on RGB information according to claim 1, it is characterized in that: In step 1, the two-dimensional color image calibration plate is composed of seven parallel gradient color strips of the same length and width, and the pixel values ​​(R, G, B) of the three color channels of RGB in the seven gradient color strips are as follows: Gradually changes from (255, 255, 0) to (255, 255, 255), gradually changes from (255, 0, 255) to (255, 255, 255), gradually changes from (0, 255, 255) to (255, 255, 255), gradually changes from (255, 0, 0) to (255, 255, 255), gradually changes from (0, 255, 0) to (255, 255, 255), gradually changes from (0, 0, 255) to (255, 255, 255) and gradually changes from (0, 0, 0) to (255, 255, 255).

3. The method for predicting exposure time of a structured light camera based on RGB information according to claim 1, characterized in that: In step 2, the minimum exposure time T in the exposure time calibration sequence is min The shortest 2D color image exposure time required by the structured light camera and the shortest exposure time of the 3D point cloud image The maximum value in the exposure time calibration sequence is T max The maximum exposure time of the 2D color image required by the structured light camera and the longest exposure time of 3D point cloud images The minimum value in .

4. The method for predicting exposure time of a structured light camera based on RGB information according to claim 1, characterized in that: The step 3 is specifically as follows: Step 3.1: According to the exposure time calibration sequence and the corresponding calibration plate two-dimensional color image sequence, calibrate to obtain the camera response function sequence of the three color channels R, G, and B; Step 3.2: Calculate the RGB three-color channel thresholds for the structured light camera to correctly capture effective three-dimensional point cloud information based on the two-dimensional color image sequence of the calibration plate and the three-dimensional point cloud image sequence of the calibration plate; Step 3.3: Use the camera response function sequence of the three RGB color channels to convert the RGB three color channel thresholds into cutoff exposures of the three RGB color channels that can correctly capture effective three-dimensional point cloud information.

5. The method for predicting exposure duration of a structured light camera based on RGB information according to claim 4, characterized in that: The step 3.2 is specifically as follows: First, generate the calibration plate 3D point cloud sequence P 3D The corresponding point cloud height binary mask matrix sequence P 3D_P , where the mask corresponding to the missing part of the point cloud information is set to 1, and the mask corresponding to the complete part of the point cloud information is set to 0; then the calibration plate two-dimensional color image sequence P 2D Each two-dimensional calibration plate color image and binary mask matrix sequence P 3D_P The binary mask matrix under the same exposure time in the image is corresponding to the binary mask matrix, and the RGB three color information of the calibration plate color image corresponding to the position with a value of 0 in the mask matrix is ​​set to 0 to obtain a new two-dimensional calibration plate color image. The images and matrices in the two sequences are processed in turn to obtain a new calibration plate two-dimensional color image sequence NP. 2D ; According to the new calibration plate two-dimensional color image sequence NP 2D Calculate the pixel value thresholds of the RGB three color channels that can correctly capture the effective 3D point cloud information.

6. The method for predicting exposure duration of a structured light camera based on RGB information according to claim 5, characterized in that: In step 3.2, according to the new calibration plate two-dimensional color image sequence NP 2D Calculate the pixel value thresholds of the RGB three color channels that can correctly capture the effective 3D point cloud information, specifically: First, the new calibration plate two-dimensional color image sequence NP 2D The effective pixel points of all the two-dimensional calibration plate color images in the effective pixel set A point ; Then count the valid pixel set A point The pixel value combination of all RGB channels and the number of its occurrences Then remove the number The valid pixels corresponding to the pixel value combination that is less than the preset valid pixel number threshold are obtained to obtain a new valid pixel set NA point ; Then calculate the new valid pixel set NA point The equivalent pixel value H corresponding to each valid pixel in RGB , the new valid pixel set NA point The equivalent pixel value H RGB The minimum effective pixel R value, G value and B value are used as the RGB three-color channel pixel value threshold Z that can correctly capture the effective 3D point cloud information. r , Z g , Z b .

7. The method for predicting exposure duration of a structured light camera based on RGB information according to claim 6, characterized in that: The equivalent pixel value H corresponding to each effective pixel RGB The calculation formula is as follows: Among them, H RGB is the equivalent pixel value corresponding to the pixel, is the R channel pixel value of the pixel point, is the G channel pixel value of the pixel point, is the pixel value of the B channel of the pixel.

8. The method for predicting exposure duration of a structured light camera based on RGB information according to claim 1, characterized in that: In step 5, converting the two-dimensional scene color image sequence into a real irradiance image using a camera response function sequence includes: For each two-dimensional scene color image in the two-dimensional scene color image sequence WP, the pixel values ​​of the corresponding color channels of each two-dimensional scene color image are converted into exposure values ​​using the camera response functions of the three channels R, G, and B respectively. The exposure values ​​are then divided by the corresponding exposure duration to obtain the two-dimensional scene relative irradiance map corresponding to the two-dimensional scene color image. After traversing and processing all the two-dimensional scene color images in the two-dimensional scene color image sequence WP, the two-dimensional scene relative irradiance map sequence EP is finally obtained. The irradiance map values ​​at the same pixel position in the two-dimensional scene relative irradiance map sequence EP are weighted according to the following formula to form the real two-dimensional scene absolute irradiance map E: Among them, EP x Represents the relative irradiance value of the pixel in the xth two-dimensional scene relative irradiance map, N is the number of pictures in the two-dimensional scene relative irradiance map sequence EP, WP x (γ) represents the pixel value of channel γ in the x-th two-dimensional scene color image in the two-dimensional scene color image sequence WP, where γ is a channel among R, G, and B; V is the weight function.

9. The method for predicting exposure duration of a structured light camera based on RGB information according to claim 1, characterized in that: In step 5, the maximum exposure time of the three-dimensional point cloud image corresponding to each pixel in the real irradiance image is calculated according to the cutoff exposure of the RGB three color channels that can correctly capture the effective three-dimensional point cloud information, including: The maximum exposure time T of the 3D point cloud image corresponding to pixel (i, j) in the exposure time matrix ij(min) Satisfies the following formula: T ij(min) =min(T ij(r) ,T ij(g) ,T ij(b) ) Among them, E ij(r) 、E ij(g) and E ij(b) They refer to the absolute irradiance of the R channel, the G channel, and the B channel of the pixel (i, j) in the real irradiance image respectively; T ij(r) , T ij(g) and T ij(b) Refers to the maximum exposure time corresponding to the R, G and B channels of pixel (i, j) respectively; X r , X g and X b They are the cutoff exposure of the R, G, and B color channels that can correctly capture effective 3D point cloud information.

10. The method for predicting exposure duration of a structured light camera based on RGB information according to claim 1, characterized in that: In step 5, the optimal exposure time of the three-dimensional point cloud image of the structured light camera is determined according to the exposure time matrix, specifically: First, all the duration values ​​in the exposure duration matrix are divided into intervals and the number of pixels falling into each interval is counted, and then the intervals are clustered according to the number of pixels in each interval; After clustering is completed, the interval lower limit with the smallest value is selected from each cluster as the candidate optimal exposure time; according to the ranking of the candidate exposure time, the exposure time is selected in turn according to the number of cameras supported.

Images