A stitching method based on dual-camera point cloud data

Abstract

The present invention relates to a stitching method based on dual-camera point cloud data, which includes the following steps. S1. Select a calibration object and convert the point cloud data of the calibration object into a template file. S2. Set up two cameras on the left and right and adjust the angle between the two cameras. S3. Place the calibration object in the overlapping area of the captured images of the two cameras and obtain two cloud maps. S4. Perform three-dimensional point cloud registration on the two cloud maps with the template file respectively, and use the point cloud data with the best match as the feature object. S5. Calculate the pose parameters of the two cameras and use the pose parameters as the calibration file. S6. Take a test object for comparison testing; if the test result shows that the data is correct, proceed to the next step; if the test result shows that the data is misaligned, return to step S3. S7. Use the calibration file as the deployment file. By stitching the dual-camera images, the blind area positions that cannot be seen by a single camera can be displayed, and the data of the two cameras can be mutually verified, effectively reducing the influence brought by noise points.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud processing, and particularly to a method for stitching point cloud data based on dual cameras. Background Art

[0002] In the prior art, when obtaining an object image, usually a single camera is used to obtain the image. However, the image obtained by the single camera will have incorrect noise data; and the single camera can only obtain the image of the object in one direction, and the obtained object features are limited. Therefore, in the actual application process, it cannot meet the requirements. Especially if the object is analyzed or detected subsequently, the limited object features will affect the accuracy. Summary of the Invention

[0003] Based on this, it is necessary to provide a method for stitching point cloud data based on dual cameras to solve the problem of incorrect noise data when using a single camera to obtain an object image in the prior art.

[0004] A method for stitching point cloud data based on dual cameras includes the following steps:

[0005] S1. Select a calibration object, obtain the point cloud data of the calibration object, and convert the point cloud data into a template file.

[0006] S2. Set up two cameras on the left and right, and adjust the angle between the two cameras so that the overlap degree of the images captured by the two cameras is greater than a preset overlap degree.

[0007] S3. Place the calibration object in the overlapping area of the images captured by the two cameras, and obtain two cloud maps through the two cameras.

[0008] S4. Perform three-dimensional point cloud registration on the two cloud maps with the template file respectively, and use the point cloud data with the best match as the feature object.

[0009] S5. Calculate the pose parameters of the two cameras based on the positions of the feature object in the two cameras, and use the pose parameters as the calibration file of the current two cameras.

[0010] S6. Take a test object, place the test object in the overlapping area of the images captured by the two cameras, and the two cameras perform tests according to the calibration file; if the test result shows that the point cloud matching data is correct, proceed to the next step; if the test result shows that the point cloud matching data is misaligned, jump to step S3.

[0011] S7. Use the calibration file as the deployment file for point cloud data stitching.

[0012] The above-mentioned stitching method based on dual-camera point cloud data can perform stitching through dual-camera images, display blind spots that cannot be seen by a single camera, and the data of the dual cameras can be mutually verified, effectively reducing the influence brought by noise and improving the measurement accuracy.

[0013] In one embodiment, in step S2, the included angle between the two cameras is set to 30°.

[0014] In one embodiment, in step S4, during the three-dimensional point cloud registration process, first use rough registration to find the rough registration parameters and record the rough registration parameters; then perform fine registration, and use the rough registration parameters as the initial values of the fine registration.

[0015] In one embodiment, in step S1, when selecting a calibration object, the calibration object has at most one axis of symmetry, and the color difference between the color of the calibration object and the background color when obtaining the point cloud data of the calibration object is greater than a preset value.

[0016] In one embodiment, the calibration object is extracted as the template file by using the method of segmenting the scene point cloud.

[0017] The present invention also discloses a stitching system based on dual-camera point cloud data, which includes a template file acquisition module, a camera setup module, a stitched cloud map acquisition module, a feature object acquisition module, a calibration file acquisition module, and a test module. The template file acquisition module is used to acquire the point cloud data of the calibration object and convert the point cloud data into a template file. The camera setup module is used to set up the left and right two cameras and adjust the included angle between the two cameras so that the coincidence degree of the pictures captured by the two cameras is greater than a preset coincidence degree. The stitched cloud map acquisition module is used to place the calibration object in the area where the pictures captured by the two cameras coincide, and acquire two cloud maps through the two cameras. The feature object acquisition module performs three-dimensional point cloud registration on the two cloud maps respectively with the template file, and takes the point cloud with the best match as the feature object. The calibration file acquisition module calculates the pose parameters of the two cameras based on the positions of the feature object in the two cameras, and takes the pose parameters as the calibration file of the current two cameras. The test module places the test object in the area where the pictures captured by the two cameras coincide, and the two cameras perform tests according to the calibration file; if the test result shows that the point cloud matching data is correct, the calibration file is used as the deployment file for point cloud data stitching; if the test result shows that the point cloud matching data is misaligned, the calibration files of the two cameras are re-acquired until the test result shows that the point cloud matching data is correct.

[0018] In one embodiment, during the three-dimensional point cloud registration process, first use rough registration to find the rough registration parameters and record the rough registration parameters; then perform fine registration, and use the rough registration parameters as the initial values of the fine registration.

[0019] In one embodiment, first, a calibration object with at most one axis of symmetry is selected, and then the calibration object is scanned by a scanning device to obtain the point cloud data of the calibration object; finally, the point cloud data is converted into a template file for storage; wherein, during the scanning process of the scanning device, the color difference degree between the color of the scanning background and the color of the calibration object is greater than a preset value.

[0020] The present invention also discloses a computer terminal, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor. The steps of the stitching method based on dual-camera point cloud data are implemented when the processor executes the program.

[0021] The present invention also discloses a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the stitching method based on dual-camera point cloud data are implemented.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] The stitching method based on dual-camera point cloud data of the present invention can stitch through dual-camera images, display blind area positions that cannot be seen by a single camera, and the data of the dual cameras can be mutually verified, effectively reducing the influence brought by noise points and improving the measurement accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flowchart of the stitching method based on dual-camera point cloud data.

[0025] Figure 2 It is a flowchart of the object reset method based on dual-camera point cloud data.

[0026] Figure 3 It is a module diagram of the stitching system based on dual-camera point cloud data. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0028] It should be noted that when a component is referred to as being "installed on" another component, it can be directly on the other component or there can also be an intermediate component. When a component is considered to be "arranged on" another component, it can be directly arranged on the other component or there may be an intermediate component at the same time. When a component is considered to be "fixed to" another component, it can be directly fixed to the other component or there may be an intermediate component at the same time.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this invention belongs. The terms used in the description of the present invention herein are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "or / and" used herein includes any and all combinations of one or more of the related listed items.

[0030] Embodiment 1

[0031] Please refer to Figure 1 , this embodiment discloses a stitching method based on dual-camera point cloud data, which includes the following steps S1-S7.

[0032] S1. Select a calibration object and obtain the point cloud data of the calibration object, and convert the point cloud data into a template file. In this embodiment, when selecting the calibration object, the calibration object has at most one axis of symmetry, and the color difference between the color of the calibration object and the background color when obtaining the point cloud data of the calibration object is greater than a preset value. The color difference refers to the degree of difference between two colors, and the preset value is determined according to experience or existing standards. For example, 30%, 40%, etc. In this embodiment, a table tennis racket can be used as the calibration object. The racket is black and the background is set to white. There is a distinct contrast between the calibration object and the background, and the handle and the racket face of the racket can be used as a direction to facilitate template recognition. Objects with such a background difference and a direction can be used as calibration objects, and during the shooting process, attention should be paid to the integrity of the point cloud of the calibration object, and the point cloud data should have as few noise points as possible.

[0033] In this embodiment, the method of segmenting the scene point cloud is used to extract the calibration object as a template file, and the data format of the template file is a general data format and does not depend on the hardware data.

[0034] S2. Set up two cameras on the left and right, and adjust the angle between the two cameras so that the overlap degree of the images captured by the two cameras is greater than a preset overlap degree. The preset overlap degree can be determined by looking up a table or based on experience, such as 80%, 90%, etc. Being greater than the preset overlap degree is to meet the overlap degree requirement. By adjusting the angle between the two cameras, the shooting angles of the two cameras and the overlap degree of the captured images can be changed. When the two cameras are set parallel, the image overlap degree is 0; when the angle between the two cameras continuously increases, the overlapping area of the images gradually gets closer from far away. In this embodiment, through testing, 30° is selected as the angle between the two cameras. Of course, in other embodiments, the angle between the two cameras can be adjusted.

[0035] S3. Place the calibration object in the overlapping area of the images captured by the two cameras, and obtain two cloud maps through the cameras. In this embodiment, the calibration object is placed at the exact center of the overlapping area of the images. The two cameras on the left and right respectively shoot the calibration object from two angles, and two cloud maps of the calibration object from the perspectives of the two cameras can be obtained. One side of the calibration object will appear in both of the two calibration object cloud maps, and the two sides of the calibration object will appear separately in the two calibration object cloud maps. For example, for a table tennis racket, taking the handle and the racket as the directions and being parallel to the placement directions of the two cameras, one side of the racket faces the cameras. At this time, both the side of the racket and the handle facing the cameras can be captured by the two cameras, but only the side of the handle and the side of the racket can be captured by the two cameras respectively.

[0036] S4. Perform 3D point cloud registration on the two cloud maps with the template file respectively, and use the point cloud data with the best match as the feature object. During the 3D point cloud registration process in this embodiment, first use NDT for rough registration to find the rough registration parameters and record the rough registration parameters; then perform ICP fine registration, and use the rough registration parameters as the initial values of the ICP fine registration. Rough registration and fine registration are conventional 3D point cloud registration methods in the prior art, and this embodiment does not elaborate on them. Rough registration and fine registration are methods used for acceleration in point cloud matching, which can improve the matching accuracy and speed.

[0037] S5. Calculate the pose parameters of the two cameras based on the positions of the feature object in the two cameras, and use the pose parameters as the calibration files of the current two cameras. The two calibration files can be verified with each other to improve the accuracy.

[0038] S6. Take the test object and place the test object in the overlapping area of the images captured by the two cameras. The two cameras perform tests according to the calibration files; if the test result shows that the point cloud matching data is correct, proceed to the next step; if the test result shows that the point cloud matching data is misaligned, jump to step S3.

[0039] S7. Use the calibration file as the deployment file for point cloud data stitching.

[0040] Taking a table tennis racket as an example, a single camera can only take pictures from the front of the racket. The handle and the side wall of the racket become blind spots and cannot be captured. Moreover, when using a single camera for testing, there is noise in the data. The stitching method based on dual-camera point cloud data in this embodiment can stitch through dual-camera images, display the blind spot positions that a single camera cannot see, and the data of the dual cameras can be mutually verified, effectively reducing the influence brought by noise and improving the measurement accuracy.

[0041] Embodiment 2

[0042] Please combine with Figure 2 , this embodiment also discloses an object reset method based on dual-camera point cloud data, which includes the following steps.

[0043] S1. Obtain the original point cloud data of the object. The original point cloud data of the object is obtained by using the stitching method based on dual-camera point cloud data as in Embodiment 1.

[0044] S2. Open the stored original point cloud data file, and use the original point cloud data as a reference cloud map. Select an area of interest (AOI) in the reference cloud map. When selecting the area of interest, it can be selected arbitrarily. In this embodiment, the point cloud data with obvious features is used as the area of interest for subsequent identification.

[0045] S3. Obtain the current point cloud data of the object. The process of obtaining the current point cloud data of the object is the same as the process of obtaining the original point cloud data of the object.

[0046] S4. Select the area with the highest coincidence degree with the area of interest in the current point cloud data, and judge whether the coincidence degree between this area and the area of interest is greater than a preset coincidence degree; if so, use this area as the target area. The position of the target area in the current point cloud data is equivalent to the position of the area of interest in the original point cloud data. Therefore, theoretically, the features of the target area should be the same as those of the area of interest. Therefore, in order to improve the accuracy and avoid misselecting the target area, a very high coincidence degree between the target area and the area of interest is required.

[0047] In this embodiment, in order to ensure that the position of the target area in the current point cloud data is equivalent to the position of the area of interest in the original point cloud data, when selecting the area of interest, it should be ensured that the area of interest is unique in the reference cloud map, that is, only one area of interest can be found in the reference cloud map. Avoid misselection when selecting the target area in the current point cloud data.

[0048] In this embodiment, if the coincidence degree between the selected area and the area of interest is less than a preset coincidence degree, then adjust the current position of the object and re-obtain the current point cloud data of the object until the condition is met.

[0049] In this embodiment, the region with the highest coincidence degree with the region of interest in the current point cloud data is identified by using the point cloud matching topography algorithm. If the coincidence degree between the selected region and the region of interest is less than a preset coincidence degree, the current position of the object is adjusted, and the current point cloud data of the object is acquired again until the condition is satisfied. The preset coincidence degree can be determined according to experience or by looking up a table, etc., such as 80%, 90%, etc. If the preset coincidence degree is not reached, it indicates that there may be defects in the current point cloud data. The current position of the object can be adjusted, and the current point cloud data of the object can be acquired again. Then, the target region is determined in the newly acquired current point cloud data.

[0050] S5. By comparing the region of interest and the target region, the displacement amount and the angle deflection amount between the region of interest and the target region are obtained; the displacement amount and the angle deflection amount between the region of interest and the target region represent the displacement value and the angle deflection value of the original point cloud data and the current point cloud data.

[0051] S6. Adjust the position of the object according to the displacement value and the angle deflection value. The specific position adjustment can be performed by existing equipment.

[0052] In this embodiment, before adjusting the position of the object, it is first determined whether the displacement value and the angle deflection value are respectively less than a preset value. The preset value is predetermined by the operator and can refer to specific accuracy requirements when setting, but it should be ensured that the preset value is sub-millimeter level. If both the displacement value and the angle deflection value are less than a preset value, it indicates that the error between the current position of the object and the original position is sub-millimeter level and meets the accuracy requirements. Therefore, there is no need to continue adjusting the position of the object. If either the displacement value or the angle deflection value is greater than a preset value, it indicates that the error between the current position of the object and the original position does not meet the accuracy requirements. Therefore, adjustment needs to be made according to the displacement value or the angle deflection value.

[0053] The sub-millimeter-level object reset method based on point cloud topography feature matching in this embodiment can measure the displacement difference and the attitude angle difference of the two placement positions at the sub-millimeter level through the topography feature matching method. When the displacement difference and the attitude angle difference are greater than the preset threshold, adjustment is performed to make the object return to the original position again.

[0054] The object reset method of this embodiment can be applied to the medical field. In the medical field, some patients undergo radiotherapy treatment of the human body. Radiotherapy treatment needs to target the lesion, and it is necessary to aim at the same position every time. The sub-millimeter-level accuracy of this embodiment can meet the requirements. During the application process, the direction of the laser ray of the treatment device is controlled to remain unchanged to treat the lesion. When treating for the first time, the original point cloud data of the lesion is first obtained through a scanning device, and the original point cloud data is processed. After filtering out the redundant interference data, the original point cloud data is saved as a file. Then, the original point cloud data of the lesion is made into a reference cloud map of the lesion, and the region of interest is selected in the reference cloud map. When treating the lesion with a laser ray subsequently, the current point cloud data of the lesion is first obtained by using the scanning device, and the region with a very high similarity to the region of interest is identified in the current point cloud data as the target region. By comparing the coordinate difference between the region of interest and the target region, the displacement value and the angle deflection value of the original point cloud data and the current point cloud data can be reflected, and thus the displacement value and the angle deflection value between the current lesion position and the original lesion position can be reflected. If the displacement value and the angle deflection value are very small, no adjustment is required and laser ray treatment can be directly performed. If the displacement value and the angle deflection value exceed a certain value, at this time, the laser ray cannot accurately point to the current lesion, and the patient's body position needs to be adjusted, and then the position of the lesion is adjusted. When adjusting, it can be achieved by adjusting the hospital bed.

[0055] Embodiment 3

[0056] Please refer to Figure 3 , this embodiment discloses a stitching system based on dual-camera point cloud data, which includes a template file acquisition module, a camera setup module, a stitched cloud map acquisition module, a feature object acquisition module, a calibration file acquisition module, and a test module. The template file acquisition module is used to acquire the point cloud data of the calibration object and convert the point cloud data into a template file. The camera setup module is used to set up two cameras on the left and right and adjust the angle between the two cameras so that the coincidence degree of the pictures captured by the two cameras is greater than a preset coincidence degree. The stitched cloud map acquisition module is used to place the calibration object in the overlapping area of the pictures captured by the two cameras and acquire two cloud maps through the two cameras. The feature object acquisition module performs three-dimensional point cloud registration on the two cloud maps with the template file respectively, and takes the point cloud with the best match as the feature object. The calibration file acquisition module calculates the pose parameters of the two cameras based on the positions of the feature object in the two cameras, and takes the pose parameters as the calibration file of the current two cameras. The test module places the test object in the overlapping area of the pictures captured by the two cameras, and the two cameras perform tests according to the calibration file; if the test result shows that the point cloud matching data is correct, the calibration file is used as the deployment file for stitching the point cloud data; if the test result shows that the point cloud matching data is misaligned, the calibration files of the two cameras are re-acquired until the test result shows that the point cloud matching data is correct.

[0057] In this embodiment, during the 3D point cloud registration process, first use NDT for rough registration to find the rough registration parameters and record them; then perform ICP fine registration, and use the rough registration parameters as the initial values for ICP fine registration.

[0058] In this embodiment, first, select a calibration object with at most one axis of symmetry, then scan the calibration object with a scanning device to obtain the point cloud data of the calibration object; finally, convert the point cloud data into a template file for storage; wherein, during the scanning process of the scanning device, the color difference between the color of the scanning background and the color of the calibration object is greater than a preset value.

[0059] This embodiment has the same beneficial effects as Embodiment 1.

[0060] Embodiment 4

[0061] An embodiment of the present invention provides a computer terminal, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the stitching method based on dual-camera point cloud data as described in Embodiment 1.

[0062] When the method of Embodiment 1 is applied, it can be applied in the form of software. For example, it can be designed as an independently running program and installed on a computer terminal. The computer terminal can be a computer, a smart phone, a control system, and other Internet of Things devices, etc. The method of Embodiment 1 can also be designed as an embedded running program and installed on a computer terminal, such as installed on a single-chip microcomputer.

[0063] Embodiment 5

[0064] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the steps of the stitching method based on dual-camera point cloud data as described in Embodiment 1.

[0065] When the method of Embodiment 1 is applied, it can be applied in the form of software. For example, it can be designed as an independently running program and stored on a computer-readable storage medium, such as a USB flash drive. By using a USB flash drive to implement the stitching method based on dual-camera point cloud data described in Embodiment 1, directly inserting the USB flash drive can enable an object to call the computer program in the USB flash drive during shooting, making the shooting of the object more accurate and reducing blind spots and noise. Through the method of Embodiment 5, it is beneficial to the popularization and application of the stitching method based on dual-camera point cloud data.

[0066] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the various technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0067] The above-described embodiments merely represent several implementation manners of the present invention. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention application shall be subject to the appended claims.

Claims

1. A stitching method based on dual-camera point cloud data, characterized in that, it includes the following steps: S1. Select a calibration object, obtain the point cloud data of the calibration object, and convert the point cloud data into a template file; S2. Set up two cameras on the left and right, and adjust the angle between the two cameras so that the overlap degree of the images captured by the two cameras is greater than a preset overlap degree; S3. Place the calibration object in the overlapping area of the images captured by the two cameras, and obtain two cloud maps through the two cameras; S4. Perform three-dimensional point cloud registration on the two cloud maps with the template file respectively, and use the point cloud data with the best match as the feature object; S5. Calculate the pose parameters of the two cameras based on the positions of the feature object in the two cameras, and use the pose parameters as the calibration file of the current two cameras; S6. Take a test object, place the test object in the overlapping area of the images captured by the two cameras, and the two cameras perform tests according to the calibration file; if the test result shows that the point cloud matching data is correct, proceed to the next step; if the test result shows that the point cloud matching data is misaligned, jump to step S3; S7. Use the calibration file as the deployment file for stitching point cloud data.

2. The stitching method based on dual-camera point cloud data according to claim 1, characterized in that, in step S2, the angle between the two cameras is set to 30°.

3. The stitching method based on dual-camera point cloud data according to claim 1, characterized in that, in step S4, during the three-dimensional point cloud registration process, first find the rough registration parameters through rough registration and record the rough registration parameters; then perform fine registration, and use the rough registration parameters as the initial values of the fine registration.

4. The stitching method based on dual-camera point cloud data according to claim 1, characterized in that, in step S1, when selecting the calibration object, the calibration object has at most one axis of symmetry, and the color difference between the color of the calibration object and the background color when obtaining the point cloud data of the calibration object is greater than a preset value.

5. The stitching method based on dual-camera point cloud data according to claim 4, characterized in that, in step S1, the calibration object is extracted as the template file by using the method of segmenting the scene point cloud.

6. A stitching system based on dual-camera point cloud data, characterized in that, it includes: A template file acquisition module, which is used to obtain the point cloud data of the calibration object and convert the point cloud data into a template file; A camera setup module, which is used to set up two cameras on the left and right and adjust the angle between the two cameras so that the overlap degree of the images captured by the two cameras is greater than a preset overlap degree; A stitched cloud map acquisition module, which is used to place the calibration object in the overlapping area of the images captured by the two cameras and obtain two cloud maps through the two cameras; A feature object acquisition module, which performs three-dimensional point cloud registration on the two cloud maps with the template file respectively, and uses the point cloud data with the best match as the feature object; A calibration file acquisition module, which calculates the pose parameters of the two cameras based on the positions of the feature object in the two cameras, and uses the pose parameters as the calibration file of the current two cameras; A test module that places a test object in the overlapping area of the captured images of two cameras, and the two cameras perform tests according to the calibration file; if the test result shows that the point cloud matching data is correct, the calibration file is used as the deployment file for point cloud data stitching; if the test result shows that the point cloud matching data is misaligned, the calibration files of the two cameras are re-obtained until the test result shows that the point cloud matching data is correct.

7. The stitching system based on dual-camera point cloud data according to claim 6, characterized in that, during the three-dimensional point cloud registration process, first, rough registration is used to find the rough registration parameters, and the rough registration parameters are recorded; then, fine registration is performed, and the rough registration parameters are used as the initial values for fine registration.

8. The stitching system based on dual-camera point cloud data according to claim 6, characterized in that, obtaining the template file of the calibration object includes the following steps: first, select a calibration object with at most one axis of symmetry, then scan the calibration object through a scanning device to obtain the point cloud data of the calibration object; finally, convert the point cloud data into a template file for storage; wherein, during the scanning process of the scanning device, the color difference between the color of the scanning background and the color of the calibration object is greater than a preset value.

9. A computer terminal, characterized in that, it includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, it realizes the steps of the stitching method based on dual-camera point cloud data according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: a computer program is stored thereon, and when the program is executed by a processor, it realizes the steps of the stitching method based on dual-camera point cloud data according to any one of claims 1 to 5.

Images