Structured light projection image interference point identification and removal method, and three-dimensional reconstruction method and system based on structured light projection

Abstract

The invention provides a structured light projection image interference point identification and removal method and a structured light projection-based three-dimensional reconstruction method and system, and relates to the technical field of digital image processing. In the interference point identification and removal method provided by the invention, interference points are identified by training an interference point identification model and utilizing the interference point identification model; compared with a threshold setting mode, the method has the advantages that the recognition speed is high, the precision is higher, interference points are removed through coverage operation, the blank of the original interference point position is filled, the precision of subsequent three-dimensional reconstruction is improved, and fine research on a target object is facilitated. According to the three-dimensional reconstruction method and system provided by the invention, the accuracy of feature analysis of the target object is improved, the method and system are suitable for the field with high object surface fineness requirements or the use measurement of the target object, and the damage to the target object caused by contact measurement can be avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and more specifically, to a method for identifying and removing interference points in a structured light projection image, and a three-dimensional reconstruction method and system based on structured light projection. Background Art

[0002] Structured light 3D measurement technology projects grating stripes onto the object to be measured using a projector through light source projection, modulates the surface of the object to form measurement stripes, and collects the measurement stripe images with a camera for decoding and phase calculation. Finally, the epipolar constraint criterion and stereo vision technology are used to obtain 3D data of the measured surface. The 3D data can be used to reconstruct the 3D model of the object, providing powerful tools and technical support for scientific research, industrial design, and cultural heritage protection.

[0003] Due to the limitations of the optical imaging mechanism of the projector and camera and the influence of image noise, the grating fringe patterns collected and captured by the camera and projector inevitably have invalid points such as shadows and background. These invalid points affect the quality of the three-dimensional data and further affect the three-dimensional reconstruction of the object.

[0004] At present, invalid points in the image can be removed by setting a threshold or calibrating a threshold. For example, a method for extracting coal lattice stripes is disclosed in the prior art. During the extraction process, a series of images are first preprocessed, and then binarization is performed with the gray value of the stripes as the threshold to remove noise and invalid lattice stripes on the edges, thereby ensuring the purity of the stripe image. However, in structured light projection applications, if the structured light projection image is subsequently used for three-dimensional reconstruction, high requirements are placed on the image quality. This method of removing invalid points by setting a threshold or calibrating a threshold may result in poor output image quality due to processing accuracy, which cannot meet the requirements of three-dimensional reconstruction. Summary of the invention

[0005] In order to solve the problem of low interference point removal accuracy in traditional image interference point removal methods, the present invention proposes a method for identifying and removing interference points in structured light projection images, which improves the accuracy of interference point removal and ensures the purity of structured light projection images. The present invention also proposes a three-dimensional reconstruction method and system based on structured light projection, which performs three-dimensional reconstruction on the basis of high-quality structured light projection images, facilitates detailed research on target objects, and provides tools and technical support for multiple fields.

[0006] To solve the above problems, the technical solution adopted in this application is:

[0007] On the one hand, the present application proposes a method for identifying and removing interference points in a structured light projection image, comprising the following steps:

[0008] S1: Acquire a structured light projection sample image, wherein the structured light projection sample image includes interference points, and label the interference points;

[0009] S2: using the labeled structured light projection sample images to train the interference point recognition model to obtain a trained interference point recognition model;

[0010] S3: Acquire a structured light stripe surface image of the target object to be identified;

[0011] S4: inputting the structured light projection image to be identified into the trained interference point recognition model, outputting label information of the structured light projection image to be identified, thereby identifying the interference point;

[0012] S5: Analyze the feature data of the normal area of ​​the structured light projection image to be identified except the interference point, and based on the feature data of the normal area, cover the interference point to remove the interference point.

[0013] In this technical solution, the method of identifying interference points by training the interference point recognition model has faster recognition speed and higher accuracy than the threshold setting method. The interference points are removed by covering operation to fill the gaps in the original interference point positions, which is conducive to the subsequent detailed study of the target object.

[0014] Preferably, after S3 and before S4, it also includes: calibrating and correcting the received structured light projection image of the target object; performing phase analysis and frequency analysis on the light stripes in the structured light stripe surface image of the target object to be identified, and extracting the coded information of the structured light to facilitate accurate calculation of the three-dimensional coordinates of the object surface.

[0015] Preferably, the feature data is: the horizontal gradient, vertical gradient and pixel value at a certain pixel point (x, y); the feature data of the normal area is extracted, and the extraction process satisfies:

[0016] G x (x,y)=H(x+1,y)-H(x-1,y)

[0017] G y (x,y)=H(x,y+1)-H(x,y-1)

[0018] Among them, G x (x,y),G y (x, y), H(x, y) represent the horizontal gradient, vertical gradient and pixel value at the pixel point (x, y) respectively.

[0019] Preferably, based on the characteristic data of the normal area, the interference points are covered, and the process of removing the interference points includes:

[0020] Let the interference point pixel be (x h ,y l ), the pixel values ​​of the four points above, below, left, and right neighbors of the interference point pixel (x h-1 ,y l )、(x h+1 ,y l )、(x h ,y l-1 )、(x h ,y l+1 ) are extracted respectively;

[0021] The average value of the column pixels in the pixel values ​​of the four points in the upper, lower, left and right neighborhoods is used as the column pixel of the interference point pixel, and the average value of the row pixels in the pixel values ​​of the four points in the upper, lower, left and right neighborhoods is used as the row pixel of the interference point pixel. The pixel values ​​of the interference points are overwritten to remove the interference points.

[0022] In a second aspect, the present application proposes a three-dimensional reconstruction method based on structured light projection, comprising the following steps:

[0023] Acquire a structured light projection image of a target object;

[0024] Using the structured light projection image interference point recognition and removal method, interference points of the structured light projection image of the target object are recognized and removed, so as to obtain a structured light projection image of the target object with the interference points removed;

[0025] Extracting data feature information of the structured light projection image of the target object with interference points removed;

[0026] Based on the data feature information, the target object is reconstructed in three dimensions to generate a three-dimensional reconstruction model of the target object.

[0027] Preferably, the three-dimensional reconstruction method based on structured light projection further includes:

[0028] Calibrate and correct the structured light projection image of the target object;

[0029] The light fringes in the structured light projection image of the target object are phase resolved and frequency analyzed to extract the coded information of the structured light.

[0030] Preferably, the three-dimensional reconstruction method based on structured light projection further includes: performing correction processing on the generated three-dimensional reconstruction model of the target object; and outputting two-dimensional images of the target object at various angles according to the corrected three-dimensional model.

[0031] Through the above technical solution, the output quality of the structured light projection image is improved.

[0032] In a third aspect, the present application also proposes a three-dimensional reconstruction system based on structured light projection, comprising:

[0033] An image acquisition module, used to acquire a structured light projection image of a target object;

[0034] An image processing module is used to process the structured light projection image of the target object, identify and remove interference points, and obtain the structured light projection image of the target object after the interference points are removed;

[0035] A feature extraction module, used to extract data feature information of a structured light projection image of a target object after removing interference points;

[0036] The 3D reconstruction module performs 3D reconstruction of the target object based on the data feature information and generates a 3D reconstruction model of the target object.

[0037] Preferably, the image processing module includes:

[0038] An image calibration and correction module receives the structured light projection image of the target object transmitted by the image acquisition module, and calibrates and corrects the received structured light projection image of the target object;

[0039] An analysis module is used to perform phase analysis and frequency analysis on light fringes in the structured light projection image of the target object to extract the coded information of the structured light;

[0040] The interference point recognition and removal module is used to recognize and remove interference points of the structured light projection image of the target object.

[0041] Preferably, the three-dimensional reconstruction system based on structured light projection further includes:

[0042] A feedback correction module is used to correct the generated three-dimensional reconstruction model of the target object;

[0043] The image output module is used to output the corrected two-dimensional images of the target object at various angles according to the corrected three-dimensional model.

[0044] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0045] The present invention proposes a method for identifying and removing interference points in structured light projection images, a three-dimensional reconstruction method and system based on structured light projection, training an interference point recognition model, and using the interference point recognition model to identify interference points. Compared with the threshold setting method, the recognition speed is fast and the accuracy is higher. The interference points are removed by covering operations to fill the blanks of the original interference point positions, which is conducive to improving the accuracy of subsequent three-dimensional reconstruction and facilitating detailed research on target objects. The three-dimensional reconstruction method and system proposed in the present invention can realize non-contact structured light projection image acquisition and image feature extraction of target objects, which can effectively avoid contact errors caused by traditional measurement methods, improve the accuracy of target object feature analysis, and this method is also suitable for fields with high requirements for object surface fineness or use measurement of target objects, and can also avoid damage to target objects caused by contact measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram showing a flow chart of a method for identifying and removing interference points in a structured light projection image proposed in an embodiment of the present invention;

[0047] Figure 2 A schematic diagram showing a flow chart of a three-dimensional reconstruction method based on structured light projection proposed in an embodiment of the present invention;

[0048] Figure 3 A first composition diagram of a three-dimensional reconstruction system based on structured light projection proposed in an embodiment of the present invention;

[0049] Figure 4 A second composition diagram of a three-dimensional reconstruction system based on structured light projection proposed in an embodiment of the present invention;

[0050] Figure 5 A third composition diagram of a three-dimensional reconstruction system based on structured light projection proposed in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0051] The accompanying drawings are only used for illustrative purposes and are not to be construed as limiting the present application;

[0052] In order to better illustrate the present embodiment, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the actual size;

[0053] It is understandable to those skilled in the art that descriptions of certain well-known contents in the drawings may be omitted.

[0054] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments;

[0055] Example 1

[0056] like Figure 1As shown, this embodiment proposes a method for identifying and removing interference points in structured light projection images, see Figure 1 , including the following steps:

[0057] S1: Acquire a structured light projection sample image, wherein the structured light projection sample image includes interference points, and label the interference points;

[0058] S2: using the labeled structured light projection sample images to train the interference point recognition model to obtain a trained interference point recognition model;

[0059] S3: Acquire a structured light stripe surface image of the target object to be identified;

[0060] S4: inputting the structured light projection image to be identified into the trained interference point recognition model, outputting label information of the structured light projection image to be identified, thereby identifying the interference point;

[0061] S5: Analyze the feature data of the normal area of ​​the structured light projection image to be identified except the interference point, and based on the feature data of the normal area, cover the interference point to remove the interference point.

[0062] In this embodiment, the structured light projection sample image refers to the use of a projector to project grating stripes onto an object, modulate the object surface to form measurement stripes, and then use the image acquisition module to collect the measurement stripe image. Therefore, when acquiring the structured light projection sample image, an image acquisition device such as a camera can be used for acquisition. In order to ensure the effectiveness of the subsequent interference point recognition model for interference point recognition, when acquiring the structured light projection sample image, the acquisition amount of the structured light projection sample image is ensured to be sufficient, and then the interference points in the structured light projection sample image are labeled. When labeling, manual labeling can be used, or machine intelligence can be used for labeling. When training the interference point recognition model, the label information carried by the structured light projection sample image is also involved in the model training process, so that the interference point recognition model has the ability to distinguish interference points after training, which can be used for subsequent interference point recognition.

[0063] In this embodiment, the interference point recognition model is a neural network model based on deep separable convolution, such as Xception or MobileNet. During the training process, if the recognition accuracy of the interference point recognition model reaches a preset accuracy, the training is terminated. At this time, a trained interference point recognition model is obtained.

[0064] Compared with the traditional method of setting thresholds or calibrating thresholds, the present embodiment adopts a method of training interference point recognition model to identify interference points, which has a faster recognition speed and higher accuracy. In addition, the interference points are removed by covering operation to fill the gaps in the original interference point positions, which is conducive to the subsequent detailed study of the target object.

[0065] In this embodiment, after S3 and before S4, it also includes: performing phase analysis and frequency analysis on the light stripes in the structured light stripe surface image of the target object to be identified, extracting the coded information of the structured light, so as to facilitate the subsequent accurate calculation of the three-dimensional coordinates of the object surface.

[0066] In this embodiment, the normal area is the area in the structured light projection image to be identified except the interference point. After the interference point is identified, the interference point can be marked in the form of a target frame, and then feature data is extracted. The feature data is: the horizontal gradient, the vertical gradient and the pixel value at a certain pixel point (x, y); the feature data of the normal area is extracted, and the extraction process satisfies:

[0067] G x (x,y)=H(x+1,y)-H(x-1,y)

[0068] G y (x,y)=H(x,y+1)-H(x,y-1)

[0069] Among them, G x (x,y),G y (x, y), H(x, y) represent the horizontal gradient, vertical gradient and pixel value at the pixel point (x, y) respectively.

[0070] In this embodiment, based on the feature data of the normal area, the interference points are covered, and the process of removing the interference points includes:

[0071] Let the interference point pixel be (x h ,y l ), the pixel values ​​of the four points above, below, left, and right neighbors of the interference point pixel (x h-1 ,y l )、(x h+1 ,y l )、(x h ,y l-1 )、(x h ,y l+1 ) are extracted respectively;

[0072] The average value of the column pixels in the pixel values ​​of the four points in the upper, lower, left and right neighborhoods is used as the column pixel of the interference point pixel, and the average value of the row pixels in the pixel values ​​of the four points in the upper, lower, left and right neighborhoods is used as the row pixel of the interference point pixel. The pixel values ​​of the interference points are overwritten to remove the interference points.

[0073] Example 2

[0074] See also Figure 2 This embodiment proposes a 3D reconstruction method based on structured light projection, comprising the following steps:

[0075] SA: Obtain a structured light projection image of the target object;

[0076] SB: using a structured light projection image interference point recognition and removal method to recognize and remove interference points in the structured light projection image of the target object, and obtaining a structured light projection image of the target object with the interference points removed;

[0077] SC: Extracts data feature information of structured light projection images of target objects with interference points removed;

[0078] SD: Based on the data feature information, the target object is reconstructed in three dimensions to generate a three-dimensional reconstruction model of the target object.

[0079] In this embodiment, the three-dimensional reconstruction method based on structured light projection further includes:

[0080] After SA acquires the structured light projection image of the target object, before performing the interference point identification and removal described in SB, the structured light projection image of the target object is calibrated and corrected; and, the light stripes in the structured light projection image of the target object are phase resolved and frequency analyzed to extract the coded information of the structured light.

[0081] In this embodiment, the process of performing three-dimensional reconstruction of the target object based on the feature data includes:

[0082] The three-dimensional coordinate data of the target object surface is obtained by using optical geometry and triangulation methods, and three-dimensional reconstruction is performed based on the data feature information.

[0083] After generating the three-dimensional reconstruction model, the three-dimensional reconstruction method based on structured light projection also includes: correcting the generated three-dimensional reconstruction model of the target object; outputting two-dimensional images of the target object at various angles based on the corrected three-dimensional model, thereby improving the output quality of the structured light projection image.

[0084] Example 3

[0085] like Figure 3 As shown, this embodiment proposes a three-dimensional reconstruction system based on structured light projection, including:

[0086] An image acquisition module, used to acquire a structured light projection image of a target object;

[0087] An image processing module is used to process the structured light projection image of the target object, identify and remove interference points, and obtain the structured light projection image of the target object after the interference points are removed;

[0088] A feature extraction module, used to extract data feature information of a structured light projection image of a target object after removing interference points;

[0089] The 3D reconstruction module performs 3D reconstruction of the target object based on the data feature information and generates a 3D reconstruction model of the target object.

[0090] like Figure 4 As shown, in the three-dimensional reconstruction system based on structured light projection proposed in this embodiment, the image processing module includes:

[0091] An image calibration and correction module receives the structured light projection image of the target object transmitted by the image acquisition module, and calibrates and corrects the received structured light projection image of the target object;

[0092] An analysis module is used to perform phase analysis and frequency analysis on light fringes in the structured light projection image of the target object to extract the coded information of the structured light;

[0093] The interference point recognition and removal module is used to recognize and remove interference points of the structured light projection image of the target object.

[0094] like Figure 5 As shown, this embodiment also proposes a 3D reconstruction system based on structured light projection. Figure 4 In addition to the structure shown, the three-dimensional reconstruction system based on structured light projection also includes:

[0095] A feedback correction module is used to correct the generated three-dimensional reconstruction model of the target object;

[0096] The image output module is used to output the corrected two-dimensional images of the target object at various angles according to the corrected three-dimensional model.

[0097] In this embodiment, the image acquisition module includes a hardware component unit such as a camera, and also includes an optical configuration unit, the optical configuration unit is used to configure the hardware parameters of the device, adjust the projection light in real time to ensure that it hits the surface of the target object within the camera field of view, and ensure the clarity and stability of the projected image at different angles and distances. The image acquisition module captures the structured light projection image of the target object, and its sending end is connected to the receiving end of the image calibration and correction module. The image calibration and correction module receives the structured light projection image of the target object transmitted by the image acquisition module, calibrates and corrects the received structured light projection image of the target object, and the sending end of the image calibration and correction module is connected to the receiving end of the analysis module. The image calibration and correction module calibrates and corrects the received structured light projection image of the target object, and after the calibration process is completed, it performs correction processing on it through its internal parameters and external parameters, wherein the calibration process may include steps such as head distortion correction and adjustment of internal and external parameters of the camera to improve the accuracy and reliability of subsequent processing, and the correction process is to calibrate the internal and external parameters of the camera to ensure that the structured light projection image has an accurate spatial position and geometric shape in the subsequent processing. The analysis module performs phase analysis and frequency analysis on the light fringes in the calibrated and corrected light projection image, extracts the coded information of the structured light, and facilitates the subsequent accurate calculation of the three-dimensional coordinates of the object surface.

[0098] The transmitting end of the analysis module is connected to the receiving end of the interference point recognition and removal module. The interference point recognition and removal module recognizes and removes interference points of the structured light projection image of the target object, obtains the structured light projection image of the target object with the interference points removed, and effectively removes invalid points and interference in the image.

[0099] The transmitting end of the interference point identification and removal module is connected to the receiving end of the feature extraction module. The feature extraction module extracts and separates the data feature information in the image. The data feature information is extracted through complex image processing algorithms, such as morphological processing, segmentation, etc., to extract key data feature information from the image. The data feature information may include local or global features such as edges, corners, and textures for subsequent 3D reconstruction. Before feature extraction, these images may be enhanced by filtering or other preprocessing steps to improve the accuracy and stability of feature extraction.

[0100] The 3D reconstruction module performs 3D reconstruction of the target object based on the data feature information and the coding information of the structured light, and generates a 3D reconstruction model of the target object. Specifically, the 3D reconstruction module uses the data feature information extracted by the feature extraction module to associate and match the parameter information corresponding to the stereo image. The stereo image can be a multi-view image collected by hardware devices such as sensors and cameras in the image acquisition module. Therefore, its parameter information includes the relative position and angle between the viewpoints, which is used to establish the depth and structure model of the scene. By comparing and analyzing the image features under different viewpoints, the depth information or parallax of each pixel in the scene is determined, thereby reconstructing the 3D structure of the target object.

[0101] The feedback correction module performs correction processing on the generated 3D reconstruction model of the target object. During the correction processing, the corresponding parameters of the 3D reconstruction model are replaced and optimized according to the real-time parameter information of the target object, which may involve adjusting the parallax, depth information or other key parameters to ensure the accuracy and consistency of the 3D reconstruction model. The 3D simulation model is updated in real time, and the optimized parameter information is integrated into the model to reflect the real state and changes of the current environment.

[0102] The image output module outputs the corrected two-dimensional images of the target object at various angles according to the corrected three-dimensional model.

[0103] Obviously, the above embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for identifying and removing interference points in a structured light projection image, characterized in that: The following steps are involved: S1: Acquire a structured light projection sample image, wherein the structured light projection sample image includes interference points, and label the interference points; S2: using the labeled structured light projection sample images to train the interference point recognition model to obtain a trained interference point recognition model; S3: Acquire a structured light stripe surface image of the target object to be identified; S4: inputting the structured light projection image to be identified into the trained interference point recognition model, outputting label information of the structured light projection image to be identified, thereby identifying the interference point; S5: Analyze the feature data of the normal area of ​​the structured light projection image to be identified except the interference point, and based on the feature data of the normal area, cover the interference point to remove the interference point.

2. The method for identifying and removing interference points in structured light projection images according to claim 1, characterized in that: After S3 and before S4, the method further includes: calibrating and correcting the received structured light projection image of the target object; performing phase analysis and frequency analysis on the light stripes in the structured light stripe surface image of the target object to be identified, and extracting the coding information of the structured light.

3. The method for identifying and removing interference points in structured light projection images according to claim 1, characterized in that: The feature data are: the horizontal gradient, vertical gradient and pixel value at a certain pixel point (x, y); the feature data of the normal area is extracted, and the extraction process satisfies: G x (x,y)=H(x+1,y)-H(x-1,y) G y (x,y)=H(x,y+1)-H(x,y-1) Among them, G x (x,y),G y (x, y), H(x, y) represent the horizontal gradient, vertical gradient and pixel value at the pixel point (x, y) respectively.

4. The method for identifying interference points in structured light projection images according to claim 3, characterized in that: Based on the characteristic data of the normal area, the interference points are covered. The process of removing the interference points includes: Let the interference point pixel be (x h ,y l ), the pixel values ​​of the four points above, below, left, and right neighbors of the interference point pixel (x h-1 ,y l )、(x h+1 ,y l )、(x h ,y l-1 )、(x h ,y l+1 ) are extracted respectively; The average value of the column pixels in the pixel values ​​of the four points in the upper, lower, left and right neighborhoods is used as the column pixel of the interference point pixel, and the average value of the row pixels in the pixel values ​​of the four points in the upper, lower, left and right neighborhoods is used as the row pixel of the interference point pixel. The pixel values ​​of the interference points are overwritten to remove the interference points.

5. A three-dimensional reconstruction method based on structured light projection, characterized in that: The following steps are involved: Acquire a structured light projection image of a target object; Using the method for identifying and removing interference points in a structured light projection image according to any one of claims 1 to 4, interference points in a structured light projection image of a target object are identified and removed to obtain a structured light projection image of the target object with the interference points removed; Extracting data feature information of the structured light projection image of the target object with interference points removed; Based on the data feature information, the target object is reconstructed in three dimensions to generate a three-dimensional reconstruction model of the target object.

6. The three-dimensional reconstruction method based on structured light projection according to claim 5, characterized in that: Also includes: Calibrate and correct the structured light projection image of the target object; The light fringes in the structured light projection image of the target object are phase resolved and frequency analyzed to extract the coded information of the structured light.

7. The three-dimensional reconstruction method based on structured light projection according to claim 6, characterized in that: Also includes: Correcting the generated three-dimensional reconstructed model of the target object; Based on the corrected 3D model, the 2D images of the target object at various angles are output.

8. A three-dimensional reconstruction system based on structured light projection, characterized in that: include: An image acquisition module, used to acquire a structured light projection image of a target object; An image processing module is used to process the structured light projection image of the target object, identify and remove interference points, and obtain the structured light projection image of the target object after the interference points are removed; A feature extraction module, used to extract data feature information of a structured light projection image of a target object after removing interference points; The 3D reconstruction module performs 3D reconstruction of the target object based on the data feature information and generates a 3D reconstruction model of the target object.

9. The three-dimensional reconstruction system based on structured light projection according to claim 8, characterized in that: The image processing module comprises: An image calibration and correction module receives the structured light projection image of the target object transmitted by the image acquisition module, and calibrates and corrects the received structured light projection image of the target object; An analysis module is used to perform phase analysis and frequency analysis on light fringes in the structured light projection image of the target object to extract the coded information of the structured light; The interference point recognition and removal module is used to recognize and remove interference points of the structured light projection image of the target object.

10. The three-dimensional reconstruction system based on structured light projection according to claim 9, characterized in that: Also includes: A feedback correction module is used to correct the generated three-dimensional reconstruction model of the target object; The image output module is used to output the corrected two-dimensional images of the target object at various angles according to the corrected three-dimensional model.

Images