Double-line laser point cloud hole filling method, device, equipment and medium

Abstract

The invention relates to the technical field of data processing, in particular to a double-line laser point cloud cavity filling method and device, equipment and a medium, and the method comprises the steps: obtaining the point cloud data of a first camera and a second camera, and enabling the first camera and the second camera to be installed in a double-line laser horizontal opposite manner; extracting feature point cloud and reference surface point cloud based on the point cloud data, and establishing a virtual coordinate system; respectively calculating transformation matrixes from the first camera and the second camera to a virtual coordinate system according to the feature point cloud and the reference surface point cloud; performing spatial transformation on the point clouds acquired by the first camera and the second camera by using the transformation matrix obtained by calculation; and fusing the point clouds transformed to the virtual space, and filling the hole point clouds. According to the invention, data holes generated by single-camera scanning can be effectively filled, and the integrity and the measurement precision of the point cloud are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a dual-line laser point cloud hole filling method, device, equipment and medium. Background Art

[0002] At present, in the fields of intelligent manufacturing, chip processing and design, medical equipment, robots, unmanned driving, etc., more and more non-contact measurement needs are showing an increasing trend. For example, applications in glue road detection, new energy battery pole detection, mobile phone case middle frame flatness detection, contour measurement, PCB board solder volume defect detection, etc. However, due to the differences in line width, single contour line points, reference installation height, point cloud resolution, and dynamic repeatability of depth values ​​of different models of line laser equipment, in some high-precision inspection projects, there are often some cases where part of the data is lost in one direction when scanning the product due to the low installation height of the camera and the small visual depth of field. If the horizontal angle of the camera is adjusted, there will be a certain deviation between the collected data and the real data.

[0003] Therefore, the technical problem of missing hole information in single-line laser point cloud in the existing technology needs to be solved urgently. Summary of the invention

[0004] The main purpose of this application is to provide a dual-line laser point cloud hole filling method, device, equipment and medium, aiming to solve the technical problem of the existing single-line laser point cloud missing the formation of hole information.

[0005] In order to achieve the above-mentioned invention object, the present application proposes a dual-line laser point cloud hole filling method, the method comprising:

[0006] Acquire point cloud data of a first camera and a second camera, wherein the first camera and the second camera are installed with dual-line lasers facing each other horizontally;

[0007] Extracting feature point clouds and reference surface point clouds based on the point cloud data, and establishing a virtual coordinate system;

[0008] Calculating transformation matrices of the first camera and the second camera to a virtual coordinate system respectively according to the feature point cloud and the reference plane point cloud;

[0009] Using the calculated transformation matrix, spatially transform the point clouds collected by the first camera and the second camera;

[0010] The point clouds transformed into the virtual space are fused to fill the empty point clouds.

[0011] Furthermore, before the step of acquiring the point cloud data of the first camera and the second camera, the step includes:

[0012] Initializing a machine system that controls the first camera and the second camera;

[0013] Determine the starting coordinates for starting acquisition according to the size of the calibration block to be scanned;

[0014] The end coordinates of the machine are set according to the end position of the acquisition to achieve the alignment of the point cloud data acquired by the first camera and the second camera in space.

[0015] Furthermore, the step of acquiring the point cloud data of the first camera and the second camera includes:

[0016] When it is recognized that the machine reaches the starting coordinates, a trigger signal is sent to the first camera and the second camera;

[0017] Based on the trigger signal, the first camera and the second camera are controlled to start the point cloud data collection, wherein during the operation of the machine, pulse signals are continuously sent to the camera according to a set pulse interval to achieve synchronous collection of point cloud data;

[0018] When the machine reaches the predetermined end coordinates, an end signal is sent to the first camera and the second camera to stop collecting point cloud data;

[0019] Acquire point cloud data collected by the first camera and the second camera.

[0020] Furthermore, the step of extracting feature point clouds and reference surface point clouds based on the point cloud data and establishing a virtual coordinate system includes:

[0021] Identify and extract point clouds having the same features as the specific feature point cloud on the calibration block from both the point cloud data collected by the first camera and the point cloud data collected by the second camera, and obtain the feature point cloud;

[0022] Selecting a preset area from the point cloud data acquired by the first camera as a first reference surface point cloud;

[0023] Selecting a preset area from the point cloud data acquired by the second camera as a second reference surface point cloud; wherein the first reference surface point cloud and the second reference surface point are located in the same plane on the calibration block;

[0024] A virtual coordinate system is established using the extracted feature point cloud, the first reference plane point cloud, and the second reference plane point cloud.

[0025] Furthermore, the step of respectively calculating the transformation matrices of the first camera and the second camera to the virtual coordinate system according to the feature point cloud and the reference plane point cloud comprises:

[0026] Calculate the feature point cloud extracted from the first camera and the feature point cloud extracted from the second camera to determine the offset value of the first camera and the second camera in the xyz direction in the virtual coordinate system;

[0027] Performing point cloud plane fitting based on the reference surface point cloud extracted by the first camera and the reference surface point cloud extracted by the second camera, and calculating the position and rotation relationship of the first camera and the second camera relative to the virtual coordinate system;

[0028] According to the offset values ​​of the first camera and the second camera in the virtual coordinate system and the rotation relationship of the xyz coordinate axis, the transformation matrices of the first camera and the second camera in the virtual coordinate system are obtained.

[0029] Furthermore, the step of calculating the feature point cloud extracted from the first camera and the feature point cloud extracted from the second camera to determine the offset value of the first camera and the second camera in the xyz direction in the virtual coordinate system includes:

[0030] Perform circle fitting on the feature point clouds extracted by the first camera and the second camera to obtain the circle center coordinates p1(x1, y1, z1) and p2(x2, y2, z2);

[0031] Applying a statistical filtering algorithm to the reference plane point cloud extracted by the first camera and the second camera to remove noise points;

[0032] The least square method is used to solve the plane equation parameters of the cleaned reference surface point cloud, and the plane equation Ax+By+Cz+d=0 is obtained, where C=1;

[0033] According to the plane equation and the coordinates of the center of the circle (x1, y1) and (x2, y2), calculate the z1 and z2 values ​​of the pair;

[0034] Calculate the offset values ​​of the two cameras in the x, y, and z directions, which are x1-x2, y1-y2, and z1-z2 respectively.

[0035] Furthermore, the step of fusing the point cloud transformed into the virtual space and filling the empty point cloud comprises:

[0036] Obtaining transformed point clouds after the point clouds collected by the first camera and the second camera are transformed;

[0037] The point cloud resolutions of the two cameras establish the x-axis and y-axis resolutions of the virtual coordinate system;

[0038] Calculate the minimum and maximum x and y values ​​of the transformed point cloud to determine the spatial range of the point cloud;

[0039] For each spatial position point p(x0, y0) in the virtual coordinate system, query the point clouds of the first camera and the second camera based on the search radius and calculate the mean of the z value;

[0040] For the point p without point cloud data in the query result, it is marked as an invalid point;

[0041] Traverse all points to complete the fusion of point clouds and fill holes.

[0042] The second aspect of the present application provides a dual-line laser point cloud hole filling device, comprising:

[0043] An acquisition module, used for acquiring point cloud data of a first camera and a second camera, wherein the first camera and the second camera are installed with dual-line lasers facing each other horizontally;

[0044] Establishing a module, for extracting a feature point cloud and a reference surface point cloud based on the point cloud data, and establishing a virtual coordinate system;

[0045] A calculation module, used to calculate the transformation matrix of the first camera and the second camera to the virtual coordinate system according to the feature point cloud and the reference plane point cloud;

[0046] A transformation module, used to perform spatial transformation on the point clouds collected by the first camera and the second camera using the calculated transformation matrix;

[0047] The fusion module is used to fuse the point cloud transformed into the virtual space and fill the hole point cloud.

[0048] The third aspect of the present application also includes a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0049] The fourth aspect of the present application also includes a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of any of the above methods when executed by a processor.

[0050] Beneficial Effects

[0051] In the field of three-dimensional point cloud data processing, this application can effectively solve the problem of data loss caused by single camera imaging and the problem of hardware defects. By using dual heads to calibrate the relationship between their installation positions, the problem of point cloud holes can be effectively solved. It can effectively fill the data holes generated by single camera scanning and improve the integrity and measurement accuracy of the point cloud. Through the dual-camera system and precise calibration, the problem of data loss caused by viewing angle limitation and occlusion is reduced, and the efficiency and accuracy of data processing are improved. The method also achieves precise alignment of point cloud data by calculating the transformation matrix, reduces morphological distortion, and enhances robustness to abnormal data. In addition, the automated data processing process reduces human errors and improves the convenience of operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic diagram of a flow chart of a dual-line laser point cloud hole filling method according to an embodiment of the present application;

[0053] Figure 2 This is a schematic structural block diagram of a dual-line laser point cloud hole filling device according to an embodiment of the present application;

[0054] Figure 3 A schematic block diagram of the structure of a computer device according to an embodiment of the present application.

[0055] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "above", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when an element is said to be "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any module and all combinations of one or more associated listed items.

[0058] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0059] Reference Figure 1 The embodiment of the present invention provides a dual-line laser point cloud hole filling method, comprising steps S1-S5, specifically:

[0060] S1. Acquire point cloud data of a first camera and a second camera, wherein the first camera and the second camera are installed with dual-line lasers facing each other horizontally;

[0061] S2, extracting feature point clouds and reference surface point clouds based on the point cloud data, and establishing a virtual coordinate system;

[0062] S3, calculating the transformation matrices of the first camera and the second camera to a virtual coordinate system respectively according to the feature point cloud and the reference plane point cloud;

[0063] S4. Using the calculated transformation matrix, spatially transform the point clouds collected by the first camera and the second camera;

[0064] S5. Fusing the point cloud transformed into the virtual space and filling the empty point cloud.

[0065] As described in step S1 above, in this embodiment, point cloud data collected by two-line laser cameras (i.e., the first camera and the second camera) are installed horizontally opposite to each other. First, in order to ensure that the two cameras can effectively cooperate and capture complete product information, the two line laser cameras need to be installed horizontally opposite to each other, and a certain installation gap is allowed between them. This installation method can ensure that when some data is lost during imaging of the product due to different heights, the missing data of one camera can be supplemented by the other camera.

[0066] Specifically, the installation of dual-line laser cameras needs to meet certain requirements. For example, determine the reference installation distance according to the camera model, and ensure that the vertical distance from the camera laser emission port to the surface of the product meets the requirements of the installation distance plus the camera depth of field. At the same time, the dual cameras need to be installed oppositely in the same horizontal plane to ensure that when scanning the same object, the object is within approximately the same depth of field. In addition, the height direction of the camera should be kept consistent as much as possible to reduce imaging deviations caused by height differences. For some products, there may be problems with cross-reflection of light during the imaging process, so reserving a certain installation gap can help eliminate this effect.

[0067] After completing the camera installation, the next step is to initialize the operation of the operation and control machine equipment. This step is crucial because it directly determines the quality of subsequent point cloud data acquisition. During the initialization process, the position of the operation and control needs to be set according to the specific parameters of the calibration block to be scanned. Normally, the long side of the calibration block is used as the scanning direction of the operation and control movement, and a certain distance is reserved from the front end of the long side of the calibration block as the starting position of the operation and control, and the end position stops at a distance that is also reserved beyond the end of the calibration block. Adjust the device to the above specified position through manual mode, and record it as the starting coordinates and end coordinates of the system operation. Doing so not only ensures stability throughout the scanning process, but also helps to improve the consistency and accuracy of the final generated point cloud data.

[0068] After the preparation is completed, you can officially start to acquire point cloud data. This process involves multiple sub-steps: first, the operation control is reset to zero, and then the control control starts to move from the preset starting position, and a trigger signal is sent to the camera when it reaches the starting position to start data acquisition. For example, the encoder plus IO trigger signal method means that during the operation control movement, pulse signals will be continuously sent to the camera according to the distance between pulses, so that the camera can acquire data according to the preset acquisition interval (such as 20 pulses triggering a photo). When the control moves to the predetermined end position, the camera ends the acquisition and stores all the collected point cloud data in the memory. It is worth noting that according to the different calibration block materials, the imaging parameters of the camera, such as exposure time, HDR settings, etc., need to be adjusted separately while collecting data to ensure the best imaging quality. The purpose of this is to deal with the imaging fluctuations that may be caused by different materials, so as to ensure that the acquired data is as stable and reliable as possible.

[0069] In summary, step S1 is not just about obtaining the raw point cloud data from the two cameras, it also includes a series of meticulous operations, including but not limited to the correct installation of the camera, the precise initialization of the control system, and the optimization of camera parameters for calibration blocks of different materials. These measures work together to lay a solid foundation for subsequent processing, ensure the quality and integrity of the point cloud data, and thus improve the effectiveness and reliability of the entire void filling method.

[0070] As described in step S2 above, based on the previously collected point cloud data of the first camera and the second camera, the positioning feature point cloud and the reference surface point cloud are extracted, and a virtual coordinate system is constructed on this basis. This process is crucial to ensure the accuracy of subsequent transformation matrix calculation and point cloud fusion. To this end, a series of carefully designed methods and technical means are required to achieve high-quality data processing.

[0071] First, for the raw point cloud data collected by each camera, the focus is on identifying and extracting those point clouds that match specific features on the calibration block. In this case, six circular hole features are selected as positioning features because they have good repeatability and are easy to detect. By setting a rectangular area (such as cam1_mask1), the entire range containing these six circular holes can be preliminarily defined as the extraction area for the positioning feature point cloud. Next, along the long side of the calibration block, another rectangular area cam1_mask2 is set approximately 20 mm below cam1_mask1 to extract the reference surface point cloud of camera 1. Since cameras 1 and 2 are installed horizontally opposite to each other, the images of the calibration block they capture are in opposite directions. Therefore, when processing the data collected by camera 2, a cam2_mask1 area is also defined as its positioning feature area, and cam2_mask2 is set approximately 20 mm above cam2_mask1 as the reference surface extraction area. This arrangement ensures that even if the two cameras have different viewing angles, they can accurately correspond to the same physical position.

[0072] Next, in order to establish a virtual coordinate system, the selection of feature point clouds needs to be further refined. Specifically, for each camera's positioning feature area (cam1_mask1 or cam2_mask1), the corresponding grayscale data is selected for segmentation, and then the image is binarized. Based on the binarized image, the sub-pixel edge point coordinates of the six circular holes can be extracted, and these coordinates are assigned to the corresponding circular holes using a clustering algorithm. Using the least squares method, the optimal circular contour of each circular hole can be fitted to obtain their center coordinates (x, y). At the same time, it is also necessary to consider how to extract valid height information from the depth map. For the two reference plane areas cam1_mask2 and cam2_mask2, all valid height values ​​are traversed, and the three-dimensional spatial coordinates (x, y, z) corresponding to each pixel are calculated according to the current camera resolution (lateral resolution is 0.02mm, longitudinal resolution is 0.02mm). This process not only involves simple mathematical operations, but also requires the exclusion of invalid coordinates without height values ​​to avoid introducing unnecessary errors.

[0073] Once enough feature point clouds and reference surface point clouds are obtained, we can start to establish a virtual coordinate system. Here, a standard spatial rectangular coordinate system is used, with its origin located at a fixed position, and the plane equation Ax+By+Cz+D=0 is used to describe the direction of the reference plane. In particular, in this example, C=1 is set, and A, B, and D are equal to zero, which means that the Z axis is always perpendicular to the XY plane. In this way, it can be ensured that all transformations occur in the same reference frame, making subsequent calculations more intuitive and accurate. In addition, considering the complexity in practical applications, the resolution differences between different cameras must also be taken into account. If the models of the two cameras are inconsistent, it is necessary to set appropriate X-axis and Y-axis scale standards according to their specific parameters so that all point cloud data can be correctly represented in the virtual coordinate system.

[0074] As described in step S3 above, based on the feature point cloud and reference surface point cloud extracted in step S2, the transformation matrix from the first camera and the second camera to the virtual coordinate system is calculated respectively. This is one of the very important links in the entire two-line laser point cloud hole filling method, because it is directly related to whether the point cloud data collected by the two cameras can be accurately converted to a unified coordinate system, thereby providing a reliable guarantee for the subsequent point cloud fusion.

[0075] In order to calculate the transformation matrix, the first thing to solve is to determine the offset values ​​of the two cameras in the three directions of XYZ. The specific method is to process the feature point clouds extracted from the first camera and the second camera. First, the least squares method is used to fit the two sets of feature point clouds respectively to solve the respective optimal circle center coordinates p1(x1, y1, z1) and p2(x2, y2, z2). It should be noted here that although the XY coordinates of the center of the circle are obtained, the Z coordinate still needs to be further determined. To this end, the statistical filtering algorithm is applied to the reference plane point cloud extracted by the two cameras to remove noise points, and then the least squares method is used to solve the plane equation parameters. The form of the plane equation is Ax+By+Cz+d=0, where C is set to 1, because it is necessary to ensure that the Z axis is always perpendicular to the XY plane. The corresponding Z coordinate value can be calculated through the plane equation and the known XY coordinates of the center of the circle. Finally, by comparing the center coordinates of the two cameras, we can get their relative offsets in the three directions of XYZ, which are x_det = x1-x2, y_det = y1-y2, and z_det = z1-z2. These offsets constitute part of the translation transformation, which is the so-called translation matrix T.

[0076] In addition to translation, the influence of rotation also needs to be considered. This involves calculating the posture changes of the two cameras relative to the virtual coordinate system. It is known that each reference plane determined by the camera has a direction vector. Assume that the plane direction vector determined by the first camera is a, the second camera is b, and the XOY reference plane direction vector in the virtual coordinate system is c. By comparing the angles between these three vectors, the rotation angles of the two cameras relative to the virtual coordinate system on the XYZ axis can be known. Specifically, the angles α_x, α_y, α_z and β_x, β_y, β_z between vectors a, b and vector c on the XYZ axis can be calculated respectively. Based on these angles, the corresponding rotation transformation matrices R1 and R2 can be constructed, which represent the rotation of the first camera and the second camera relative to the virtual coordinate system, respectively. Finally, the translation matrix T is combined with the rotation transformation matrix to form the complete transformation matrices M1 and M2, which are used to map the original point cloud data to the virtual coordinate system.

[0077] It is worth noting that in actual operation, when the models or resolutions of the two cameras are different, the calculation method needs to be adjusted accordingly. In addition, due to the various interference factors that may exist in the actual environment, such as vibration, temperature change, etc., this may also cause slight deformation or displacement of the point cloud data. Therefore, when calculating the transformation matrix, we should not only pursue theoretical accuracy, but also consider the robustness and adaptability in actual situations. In order to achieve this goal, some optimization measures are introduced in the calculation process, such as using weighted least squares instead of ordinary least squares to improve the robustness of the calculation results; at the same time, the original data will be iterated multiple times to ensure that each transformation is as close to the actual situation as possible.

[0078] As described in step S4 above, the point clouds collected by the first camera and the second camera are spatially transformed using the calculated transformation matrices. This process converts the point cloud data originally scattered in their respective coordinate systems into a unified virtual coordinate system for subsequent efficient point cloud fusion. To achieve this, a series of rigorous and meticulous operations are required to ensure that each step can be completed accurately and without error.

[0079] First, let's review the results obtained in step S3: For the first camera and the second camera, their respective translation matrices T and translated rotation transformation matrices R1 and R2 are calculated respectively. These two matrices together determine how to transform the original point cloud data from the camera's own coordinate system to the virtual coordinate system. In practical applications, since the installation position relationship of the two cameras has been accurately measured and reflected in the transformation matrix, it is only necessary to apply the translation matrix to the data of one of the cameras. In this case, the translation matrix T is selected to be applied to the original point cloud data of the first camera to adjust its position to the position corresponding to the second camera. Next, the translated point cloud data of the first camera is multiplied by its rotation transformation matrix R1, thereby achieving a complete transformation of the point cloud data of the first camera. The same operation is also applicable to the second camera, except that its rotation transformation matrix R2 is directly used for transformation without additional translation operation. This is because the relative position relationship of the two cameras has been corrected by the translation matrix T, so that they can be compared and processed in the same virtual coordinate system.

[0080] However, it is not enough to just complete the spatial transformation of point cloud data. The characteristics and potential problems of the point cloud data itself also need to be considered. For example, due to the height difference and occlusion on the product surface, line laser acquisition may form some invalid data. These invalid data not only cannot participate in the transformation calculation, but may interfere with the normal point cloud fusion process. Therefore, before the transformation, the original point cloud data needs to be preprocessed to screen out valid point clouds. Specifically, it will be checked whether each point cloud falls within a reasonable depth range, whether there are abnormally high noise points, or whether there are obvious discontinuities. Only valid point clouds that have been strictly screened will be included in the transformation calculation. In addition, in order to ensure the consistency and accuracy of the transformation results, the quality of the transformed point cloud data will also be evaluated, such as checking the degree of overlap between the point clouds before and after the transformation, the distribution density and other factors. If any abnormal phenomenon is found, the transformation parameters should be adjusted in time until a satisfactory effect is obtained.

[0081] As the point cloud data is gradually converted to the virtual coordinate system, a problem is noticed: since the two cameras shoot the same object from different angles, there may be overlap of point clouds in some areas after the transformation. This situation will not only not affect the final point cloud fusion, but also provide more sources of information. In order to make full use of this redundant information, special attention is paid to the processing of overlapping areas during the transformation process. For example, a search radius r can be set to perform a local query with a certain point as the center in the transformed virtual coordinate system to find all nearby point clouds from the first camera and the second camera. Then, based on the distribution characteristics of these point clouds, such as average distance, density difference, etc., it can be judged whether they belong to different observations of the same physical location. If so, weighted averaging or other aggregation strategies can be adopted to merge these point clouds into a single, more accurate point cloud representation. Doing so can not only reduce redundancy, but also significantly improve the quality of the final point cloud.

[0082] As described in step S5 above, after completing the spatial transformation of the point cloud, the next key step is to fuse the point cloud data transformed to the virtual coordinate system to effectively fill the holes that may appear in the original scanning process, and ensure that the final generated three-dimensional model is as complete and accurate as possible. Specifically, when the point clouds of two cameras are converted to the same virtual coordinate system, multiple point clouds from different cameras may overlap at certain coordinate positions, that is, redundant point clouds are formed. In order to deal with this situation and achieve effective fusion, the resolution standard of the virtual coordinate system is first determined based on whether the two cameras are of the same model: if the models are the same, the camera's own resolution is directly used; if they are inconsistent, a suitable resolution is set as the standard for the x-axis and y-axis scales. Next, the minimum x, maximum x, minimum y, and maximum y values ​​of the transformed point cloud are calculated to define the spatial range of the entire point cloud, and memory is opened up for storing the fused point cloud data. Then, based on the above-opened memory, each coordinate position (p(x0,y0)) is traversed one by one according to the row and column conditions, and a search radius r is defined for each coordinate position. Within this radius, the corresponding point cloud data is queried from the transformed point clouds of camera 1 and camera 2 respectively. For each queried point cloud set (n1 from camera 1, n2 from camera 2), all z values ​​are added to get the sum z1 and z2, and then divided by the respective number n1 and n2 to get the average z3 and z4. Finally, the average of these two averages is calculated as the fused z value z = (z3 + z4) / 2 of the coordinate position. If there is no valid point cloud at a certain position (that is, n1 and n2 are both 0), it is marked as an invalid point, indicating that there is no available point cloud data at that position. By looping through each coordinate position in the fusion memory, the above process is repeated until the fusion of the entire point cloud is completed. The point cloud data after fusion processing is more continuous and smooth, which reduces the void problem in the original data and improves the quality and accuracy of 3D reconstruction. This method not only solves the void point cloud problem that may be generated when a single line laser camera scans a product, but also improves the reliability and accuracy of the detection results. It is suitable for intelligent manufacturing, chip processing design, medical devices and other fields. It emphasizes the balance between algorithm efficiency and calculation accuracy, aiming to provide a fast and accurate solution.

[0083] Based on steps S1-S5, an example is given. For example, because camera 1 and camera 2 are installed horizontally opposite to each other, the directions of the calibration blocks in the original imaging data of camera 1 and camera 2 are opposite. Therefore, in the original point cloud data collected by camera 2, cam2_mask1 is set as the positioning feature area of ​​camera 2, and cam2_mask2 is set 20 mm above the cam2_mask1 area as the reference plane extraction area of ​​camera 2.

[0084] The virtual coordinate system in this application is established using a standard space coordinate system, the origin of which is p0(0,0,0), and the equation of the xoy plane in the coordinate system is A0x+B0y+C0z+D0=0; wherein A0=0, B0=0, C0=1, D0=0; as a preferred technical solution, the above-mentioned two-line laser point cloud hole filling method based on the calibration block includes the following features, wherein the transformation matrix is ​​calculated based on the obtained feature point clouds of camera 1 and camera 2 and the reference surface point cloud: the feature point clouds of camera 1 and camera 2 are calculated to determine the offset values ​​of the two camera coordinate systems, and the point clouds of the reference surfaces of camera 1 and camera 2 are calculated to determine the transformation relationship between the postures of the two cameras, including:

[0085] After extracting the positioning feature point clouds and reference plane point clouds of the above-mentioned cameras 1 and 2, it is necessary to calculate the camera installation relationship based on the extracted point clouds; for the positioning area cam1_mask1 determined in the above-mentioned camera 1, select the corresponding grayscale data for segmentation, and then binarize the image, and extract the sub-pixel edge point pixel coordinates of the 6 circular holes based on the data after binarization. Based on the clustering thinking, the extracted sub-pixel coordinates are segmented into the edge point coordinates corresponding to the 6 circular holes, and then the least squares method is used to calculate the center and radius of the 6 circular holes respectively. At this time, the center coordinates of the 6 circular holes obtained only have x and y coordinates. Using the reference plane area cam1_mask2 determined in camera 1, the height value in the cam1_mask2 area is extracted in the corresponding depth map, and the point cloud coordinates of each pixel are calculated according to the resolution of the current camera. In this application, the lateral resolution of the camera is x_ratio=0.02mm, and the longitudinal resolution is y_ratio=0.02mm; for coordinates where the height value does not exist, they are not involved in the calculation. The point cloud coordinate calculation rule for each pixel is: the lateral coordinate of the current pixel is x0, and the longitudinal coordinate is y0, then the corresponding point cloud coordinates in three-dimensional space are: x=x0*x_ratio; y=y0*y_ratio. Loop through all valid height values ​​in cam1_mask2 and calculate the corresponding point cloud coordinates. Determine the least squares reference plane based on the calculated valid point cloud data, and get the reference plane equation in camera 1 as plane1=A0x+B0y+C0z+D0=0. Because the center of the circular hole must be on the plane determined by cam1_mask2, substitute the x and y values ​​of the center coordinates of the 6 circular holes calculated above into the plane equation of plane1 to obtain the point cloud coordinates of the centers of the 6 circular holes, which are p11(x1,y1,z1), p12(x2,y2,z2), p13(x3,y3,z3), p14(x4,y4,z4), p15(x5,y5,z5), and p16(x6,y6,z6).As in the calculation process of camera 1 above, the same operation is performed on the grayscale data corresponding to cam2_mask1 of camera 2 to obtain the x and y coordinates of the centers of the 6 circular holes in camera 2, and then the depth data is converted to point cloud data for the cam2_mask2 reference plane area in camera 2. If the models and resolutions of the two line lasers are different, each camera needs to convert the depth data to point cloud data according to its own resolution. In this application, the two cameras have the same model, so the resolutions of the two cameras are the same. According to the point cloud data converted by camera 2, the depth data is converted to point cloud data. Calculate the reference plane of camera 2 and get the plane equation of the camera's reference plane plane2=A1x+B1y+C1z+D1=0. Then substitute the center coordinates of the 6 circular holes obtained in camera 2 into the plane equation of plane2 to get the point cloud coordinates of the centers of the 6 circular holes in the calibration block of camera 2, which are p21(x1,y1,z1), p22(x2,y2,z2), p23(x3,y3,z3), p24(x4,y4,z4), p25(x5,y5,z5), and p26(x6,y6,z6).

[0086] The calculation method of the offset component in the transformation matrix of the two cameras in this application is: the point cloud coordinates of the centers of the 6 circular holes in camera 1 are p11(x1, y1, z1), p12(x2, y2, z2), p13(x3, y3, z3), p14(x4, y4, z4), p15(x5, y5, z5), and p16(x6, y6, z6). The coordinates of the above 6 point clouds are averaged to obtain the normalized center coordinates of the 6 circular holes in camera 1 as cam_p1, where cam_p1.x is (x1+x2+x3+x4+x5+x6) / 6, cam_p1.y is (y1+y2+y3+y4+y5+y6) / 6, and cam_p1.z is z1+z2+z3+ z4+z5+z6 / 6. Similarly, according to the coordinates of the 6 circular holes in camera 2, the normalized center coordinates of the 6 circular holes in camera 2 can be obtained, cam_p2, where cam_p2.x is (x1+x2+x3+x4+x5+x6) / 6, cam_p2.y is (y1+y2+y3+y4+y5+y6) / 6, and cam_p2.z is z1+z2+z3+z4+z5+z6 / 6. Therefore, the actual physical position of camera 1 and camera 2 has an offset of x_det on the x-axis of cam_p1.x-cam_p2.x, an offset of y_det on the y-axis of cam_p1.y-cam_p2.y, and an offset of z_det on the z-axis of cam_p1.z-cam_p2.z. The final translation matrix m_t is:

[0087]

[0088] From the reference plane equations of camera 1 and camera 2 obtained above, we can know that the direction vector of the plane determined by camera 1 is a(A0, B0, C0), and the direction vector of the plane determined by camera 2 is b(A1, B1, C1). According to the direction vector of the xoy reference plane in the virtual coordinate system is a(0, 0, 1), we can calculate the angles between vector a, vector b and vector c between the x-axis, y-axis and z-axis respectively. The angles between vector a and the x-axis, y-axis and z-axis are α0, β0, γ0 respectively, and the angles between vector b and the x-axis, y-axis and z-axis are α1, β1, γ1 respectively. After sorting, we can get the transformation matrices of camera 1, camera 2 and the x-axis, y-axis and z-axis respectively, as follows:

[0089] The rotation transformation matrices of camera 1 and the x-axis, y-axis, and z-axis are:

[0090]

[0091] The transformation matrices of camera 2 and the x-axis, y-axis, and z-axis are:

[0092]

[0093] As a preferred technical solution, the above-mentioned two-line laser point cloud hole filling method based on the calibration block includes the following characteristics: the matrix transformation of the point cloud collected by camera 1 and camera 2 according to the calculated transformation matrix of camera 1 and camera 2 to the virtual coordinate system includes:

[0094] As mentioned above, the transformation matrices of cameras 1 and 2 respectively with respect to the three coordinate axes, and the translation transformation matrix between the two cameras are obtained, and the original point cloud data collected by cameras 1 and 2 are subjected to matrix transformation. In this application, the point cloud data of cameras 1 and 2 are first subjected to translation matrix operation. Because the translation matrix in this application is obtained based on the positional relationship between the two cameras, it is only necessary to multiply the data of one of the cameras by the translation matrix. In this application, the original data of camera 1 is multiplied by the average matrix m_t, and the data after translation of camera 1 is multiplied by the three transformation matrices determined by camera 1, and finally the point cloud data transformed to the virtual coordinate system is obtained. Similarly, the original data of camera 2 is multiplied by the three transformation matrices determined by camera 2 to obtain the point cloud data transformed to the virtual coordinate system.

[0095] As a preferred technical solution, the above-mentioned dual-line laser point cloud hole filling method based on the calibration block includes the following characteristics: the point cloud data of camera 1 and camera 2 are transformed into the point cloud of the virtual coordinate system for fusion, and the processing of the hole point cloud is completed, including:

[0096] The original point cloud data of camera 1 and camera 2 are converted into point cloud data in the virtual coordinate system through their corresponding transformation matrices. By analyzing the point cloud data converted into the virtual coordinate system, it can be seen that multiple point clouds will appear for the same coordinate position in the virtual coordinate system, that is, redundant point clouds are formed.

[0097] The redundant point clouds appearing in the virtual coordinate system need to be processed to solve the redundancy. The specific calculation of this application is as follows:

[0098] The cameras used in this application are of the same model, so the parameters of the cameras are the same. Therefore, a consistent strategy is adopted for the x-axis and y-axis resolutions of the virtual coordinate system and the resolution of the camera itself. If the parameters of the two cameras are inconsistent, appropriate resolutions can be set according to the specific parameters of the two cameras as the x-axis and y-axis scale standards for constructing the virtual coordinate system.

[0099] The specific processing steps of redundant point clouds are as follows:

[0100] First, for the point cloud data of camera 1 and camera 2 after their corresponding transformation matrices, calculate the minimum x, maximum x, minimum y, and maximum y of the current redundant point cloud, and open up memory according to the current calculation results and the x-axis and y-axis scales of the virtual coordinate system for the storage of the point cloud after fusion; based on the above-opened memory, traverse and calculate according to the row and column conditions. The development of memory in this application is based on the point cloud after transformation, so there will be no missing data when traversing according to the developed memory. The specific calculation and measurement are as follows: For the calculation of the z value of the point p (x0, y0), first set the search radius to r. According to the search radius, query the transformed point cloud of camera 1 with point p as the center and r as the radius to obtain the query point cloud of camera 1. Similarly, query the transformed point cloud of camera 2. The cloud is queried to obtain the query point cloud of camera 2; based on the query point cloud of camera 1 and the query point cloud of camera 2 obtained by the search, point cloud fusion processing is performed, wherein the number of query point clouds of camera 1 is n1, and the number of query point clouds of camera 2 is n2; all z values ​​of the query point clouds of camera 1 and camera 2 are added together to obtain z1 and z2; z1 is divided by n1 to obtain the mean z3 of the camera query point cloud, and z2 is divided by n2 to obtain the mean z4 of the camera query point cloud. Finally, the z of the fused point cloud of point p is (z3+z4) / 2. If the number of n1 and n2 is both zero, this application defines it as the current point p is an invalid point; loop through each coordinate position in the fusion memory, calculate the corresponding fusion z value respectively, and complete the final fusion of the transformed point cloud of camera 1 and the transformed point cloud of camera 2.

[0101] The beneficial effects of the present invention are: in the field of three-dimensional point cloud data processing, it can effectively solve the problem of data missing caused by single camera imaging and the problem of hardware defects. By using a dual head to calibrate the relationship between their installation positions, the problem of point cloud holes can be effectively solved.

[0102] In one embodiment, before the step of acquiring the point cloud data of the first camera and the second camera, the step includes:

[0103] S10, initializing a machine system for controlling the first camera and the second camera;

[0104] S11, determining the starting coordinates for starting acquisition according to the size of the calibration block to be scanned;

[0105] S12. Setting the end coordinates of the machine according to the end position of the acquisition, so as to achieve the alignment of the point cloud data acquired by the first camera and the second camera in space.

[0106] In this embodiment, first, before starting to collect point cloud data, the operation and control system is initialized. This process ensures that the two line laser cameras are installed opposite to each other in the same horizontal plane, and the installation gap between the cameras is appropriate to avoid cross reflection of light affecting the imaging quality. At the same time, the height of the camera is adjusted to ensure that the distance from the line laser emission port to the surface of the calibration block is within the depth of field range of the current camera model. In addition, initialization also includes setting the calibration block parameters required for different projects, and configuring the position of the operation and control according to the specific calibration block size. Usually, the long side of the calibration block is used as the scanning direction of the operation and control movement, and a certain distance is reserved as the starting position and the end position. Based on the actual size of the calibration block, the starting coordinates of the operation and control system are determined. Specifically, the operation and control equipment is adjusted to manual mode, and the operating equipment moves to a position where a certain distance (for example, 20mm) is reserved at the front end of the calibration block, and the current position is recorded as the starting position. This step ensures that during the formal acquisition process, the operation and control system can start scanning from the correct position, thereby ensuring that the collected data is accurate. Similarly, according to the size of the calibration block and actual needs, the end coordinates of the operation and control system are set. When the control runs along the length of the calibration block and stops at a distance beyond the end of the calibration block (also 20 mm), the current position is recorded as the end position. In this way, it can ensure that the two cameras work synchronously during the entire scanning process, and the acquired point cloud data can be correctly aligned in space, providing a solid foundation for subsequent data processing.

[0107] In one embodiment, the step of acquiring point cloud data of the first camera and the second camera includes:

[0108] S20, when it is recognized that the machine reaches the starting coordinates, sending a trigger signal to the first camera and the second camera;

[0109] S21, based on the trigger signal, controlling the first camera and the second camera to start point cloud data acquisition, wherein during the operation of the machine, pulse signals are continuously sent to the camera according to a set pulse interval to achieve synchronous acquisition of point cloud data;

[0110] S22, when the machine reaches the predetermined end coordinate, sending an end signal to the first camera and the second camera to stop collecting point cloud data;

[0111] S23: Acquire point cloud data collected by the first camera and the second camera.

[0112] In this embodiment, when the operation control machine recognizes that it has reached the preset starting coordinates, the system automatically sends a trigger signal to the first camera and the second camera. This trigger signal serves as an acquisition instruction to notify the two cameras to prepare to start the acquisition of point cloud data. In order to ensure that the two cameras can start synchronously, the design of the trigger signal needs to take into account the camera response time and any possible delay. After receiving the trigger signal, the first camera and the second camera immediately start the point cloud data acquisition process. During the operation of the operation control machine, the system continuously sends pulse signals to the camera according to the set pulse interval (for example, every time it moves a certain distance or time interval). These pulse signals act as a synchronization mechanism, ensuring that the two cameras collect data at the same frequency throughout the scanning process, thereby achieving synchronous acquisition of point cloud data. This synchronization is crucial for subsequent data alignment and fusion, and helps to reduce data deviations that may be caused by asynchrony. When the operation control machine reaches the predetermined end coordinates, the system sends an end signal to the first camera and the second camera. After receiving the end signal, the two cameras will stop further data acquisition activities. This step ensures that the acquisition process is completed within the predetermined range, avoids data collection beyond the target area, and maintains the relevance and validity of the data. The last step is to obtain the acquired point cloud data from the first camera and the second camera. This usually involves transferring the data in the camera memory to a host computer or other storage device for subsequent data processing, analysis, and application. In this process, it is necessary to ensure the integrity and accuracy of data transmission to prevent data loss or corruption. Through the above steps, this method ensures that the point cloud data acquired from the dual-line laser camera is highly synchronized in time and accurately aligned in space, providing a solid foundation for subsequent advanced processing such as feature extraction, transformation matrix calculation, and point cloud fusion. This process not only improves the efficiency of data acquisition, but also enhances the quality and reliability of the final 3D reconstruction results.

[0113] In one embodiment, the step of extracting a feature point cloud and a reference plane point cloud based on the point cloud data and establishing a virtual coordinate system includes:

[0114] S30, identifying and extracting point clouds having the same features as the specific feature point cloud on the calibration block from the point cloud data collected by the first camera and the point cloud data collected by the second camera, to obtain the feature point cloud;

[0115] S31, selecting a preset area from the point cloud data acquired by the first camera as a first reference plane point cloud;

[0116] S32, selecting a preset area from the point cloud data acquired by the second camera as a second reference plane point cloud; wherein the first reference plane point cloud and the second reference plane point cloud are located in the same plane on the calibration block;

[0117] S33: Establish a virtual coordinate system using the extracted feature point cloud, the first reference plane point cloud, and the second reference plane point cloud.

[0118] In this embodiment, point clouds with the same features as specific features (such as circular holes) on the calibration block are identified and extracted from the point cloud data collected by the first camera and the second camera. To achieve this, a rectangular area (such as cam1_mask1 for the first camera and cam2_mask1 for the second camera) is usually set, which covers all expected feature points in the calibration block. These feature point clouds can be accurately segmented through image processing techniques such as binarization and sub-pixel edge detection. The feature point cloud finally obtained will serve as the basis for subsequent calculation of the transformation matrix. From the point cloud data obtained by the first camera, a preset area is selected as the first reference surface point cloud. This area is usually a flat and easily identifiable plane part on the calibration block. For example, cam1_mask2 is set at an interval of 20mm below cam1_mask1 to ensure that a stable and flat surface area is selected. The selection of this reference surface is very important for the subsequent determination of the relative position relationship between the two cameras. Similarly, from the point cloud data obtained by the second camera, a preset area is selected as the second reference surface point cloud. Since the cameras are mounted horizontally opposite to each other, the reference plane area is located 20mm above cam2_mask1 (i.e. cam2_mask2) in the data of the second camera. It is important that the two reference planes must be located on the same plane on the calibration block, although their specific positions and the number of extracted point clouds can be different. This consistency ensures that the data acquired at different viewing angles can be accurately aligned. A virtual coordinate system is constructed using the above-mentioned extracted feature point clouds and the first and second reference plane point clouds. The virtual coordinate system adopts the form of a standard spatial coordinate system, and its origin and the direction of each axis need to be clearly defined. Specifically, the position offset and attitude rotation relationship of the two cameras relative to the virtual coordinate system can be calculated by fitting the center coordinates and the reference plane equation by the least squares method. This step not only provides a reference frame for subsequent spatial transformations, but also ensures that data from different cameras can be effectively fused and compared in a unified coordinate system. In order to ensure the accuracy and reliability of the feature point cloud and the reference surface point cloud, grayscale data segmentation, binarization processing, and clustering-based calculation of sub-pixel edge point coordinates were used for feature point cloud extraction; while for the reference surface point cloud, a statistical filtering algorithm was applied to remove noise points, and the least squares method was used to solve the plane equation parameters. In addition, considering the possible resolution differences between cameras, when the two camera models are different, the scale standard of the virtual coordinate system needs to be adjusted according to their actual resolutions to ensure the consistency of the fusion results.

[0119] In one embodiment, the step of respectively calculating the transformation matrices of the first camera and the second camera to the virtual coordinate system according to the feature point cloud and the reference plane point cloud comprises:

[0120] S40, calculating the feature point cloud extracted from the first camera and the feature point cloud extracted from the second camera, and determining the offset value of the first camera and the second camera in the xyz direction in the virtual coordinate system;

[0121] S41, performing point cloud plane fitting according to the reference surface point cloud extracted by the first camera and the reference surface point cloud extracted by the second camera, and calculating the position and rotation relationship of the first camera and the second camera relative to the virtual coordinate system;

[0122] S42: Obtain transformation matrices of the first camera and the second camera in the virtual coordinate system according to the offset values ​​of the first camera and the second camera in the virtual coordinate system and the rotation relationship of the xyz coordinate axes.

[0123] In this embodiment, the feature point cloud extracted from the first camera and the second camera is processed. First, the feature point cloud is fitted using a circle fitting algorithm to obtain the center coordinates p1 (x1, y1, z1) and p2 (x2, y2, z2) of the feature point clouds of the two cameras. Next, a statistical filtering algorithm is applied to remove noise points in the reference plane point cloud from the first camera and the second camera to ensure the accuracy of subsequent calculations. The plane equation parameters are solved for the cleaned reference plane point cloud using the least squares method to obtain the plane equation Ax+By+Cz+D=0, where C=1, in order to simplify the calculation. According to the above plane equation and the center coordinates (x1, y1) and (x2, y2), the corresponding z1 and z2 values ​​are calculated to determine the offset values ​​Δx=x1-x2, Δy=y1-y2, Δz=z1-z2 of the two cameras in the xyz direction in the virtual coordinate system. Using the reference surface point cloud data of the first camera and the second camera, the weighted least square method or other suitable plane fitting method is used for fitting to calculate the attitude change (i.e., rotation angle) of each camera relative to the virtual coordinate system. This step involves calculating the rotation angle in three dimensions (x, y, and z axes), which is usually achieved by solving the rotation matrix or quaternion. The calculation result includes not only the translation information (the offset value provided by step S40), but also the rotation information, which fully describes the relative position and attitude of the first camera and the second camera relative to the virtual coordinate system. The offset values ​​Δx, Δy, Δz of the first camera and the second camera in the virtual coordinate system obtained in step S40, and the rotation relationship relative to the virtual coordinate system calculated in step S41 are combined to construct the transformation matrix of each camera. The transformation matrix is ​​a 4x4 matrix that combines translation and rotation information and is used to transform the point cloud data in the original coordinate system to the virtual coordinate system.

[0124] The transformation matrix is ​​of the following form: Where R is a 3x3 rotation matrix and t is a 3x1 translation vector. For the first camera and the second camera, construct their corresponding transformation matrices T1 and T2 respectively.

[0125] This embodiment not only considers the spatial offset between the two cameras, but also analyzes in detail their posture changes relative to the virtual coordinate system, ensuring that the transformation matrix generated in the end can accurately reflect the geometric relationship between the two cameras and the virtual coordinate system. The application of this transformation matrix is ​​crucial for subsequent spatial transformation, point cloud fusion and hole filling, providing a solid technical foundation and ensuring the quality and accuracy of 3D reconstruction.

[0126] In one embodiment, the step of calculating the feature point cloud extracted from the first camera and the feature point cloud extracted from the second camera to determine the offset value of the first camera and the second camera in the xyz direction in the virtual coordinate system includes:

[0127] S50, performing circle fitting on the feature point clouds extracted by the first camera and the second camera to obtain circle center coordinates p1 (x1, y1, z1) and p2 (x2, y2, z2);

[0128] S51, applying a statistical filtering algorithm to the reference plane point cloud extracted by the first camera and the second camera to remove noise points;

[0129] S52, using the least square method to solve the plane equation parameters for the cleaned reference plane point cloud, and obtaining the plane equation Ax+By+Cz+d=0, where C=1;

[0130] S53, calculating the z1 and z2 values ​​of the pair according to the plane equation and the coordinates of the center of the circle (x1, y1) and (x2, y2);

[0131] S54, calculate the offset values ​​of the two cameras in the x, y, and z directions, which are x1-x2, y1-y2, and z1-z2 respectively.

[0132] In this embodiment, the feature point cloud extracted from the first camera and the second camera is processed. First, the feature point cloud is fitted using a circle fitting algorithm to obtain the center coordinates p1 (x1, y1, z1) and p2 (x2, y2, z2) of the feature point clouds of the two cameras. Next, a statistical filtering algorithm is applied to remove noise points in the reference plane point cloud from the first camera and the second camera to ensure the accuracy of subsequent calculations. The plane equation parameters are solved for the cleaned reference plane point cloud using the least squares method to obtain the plane equation Ax+By+Cz+D=0, where C=1, in order to simplify the calculation. According to the above plane equation and the center coordinates (x1, y1) and (x2, y2), the corresponding z1 and z2 values ​​are calculated to determine the offset values ​​Δx=x1-x2, Δy=y1-y2, Δz=z1-z2 of the two cameras in the xyz direction in the virtual coordinate system.

[0133] In one embodiment, the step of fusing the point cloud transformed into the virtual space and filling the hole point cloud includes:

[0134] S60, obtaining a transformed point cloud after the point cloud collected by the first camera and the second camera is transformed;

[0135] S61, the point cloud resolutions of the two cameras establish the x-axis and y-axis resolutions of the virtual coordinate system;

[0136] S62, calculating the minimum and maximum x, y values ​​of the transformed point cloud, and determining the spatial range of the point cloud;

[0137] S63, for each spatial position point p (x0, y0) in the virtual coordinate system, query the point clouds of the first camera and the second camera based on the search radius, and calculate the average of the z values;

[0138] S64. For a point p without point cloud data in the query result, mark it as an invalid point;

[0139] S65. Traverse all points to complete the fusion of point clouds and hole filling.

[0140] In one embodiment, the point cloud data collected by the first camera and the second camera after the transformation matrix conversion is obtained. These transformed point clouds have been spatially aligned in the virtual coordinate system and are ready to enter the fusion stage. According to the point cloud resolution of the two cameras, the x-axis and y-axis resolutions of the virtual coordinate system are established. If the two cameras are of the same model, the camera's own resolution is directly used as the standard for constructing the virtual coordinate system; if the camera parameters are inconsistent, the appropriate resolution is set according to the specific situation to ensure the consistency and accuracy of the fusion result. The minimum and maximum x and y values ​​of the transformed point cloud are calculated to determine the spatial range of the entire point cloud. This step helps to clarify the data area to be processed and provides a basis for subsequent memory development. For each spatial position point p (x0, y0) in the virtual coordinate system, a search radius r is set. According to this search radius, the corresponding point cloud data is queried in the transformed point clouds of the first camera and the second camera respectively. For each query result, the mean z value of all valid point clouds is calculated. Specifically, it is assumed that n1 point clouds are queried from the first camera, and the sum of their z values ​​is z1; n2 point clouds are queried from the second camera, and the sum of their z values ​​is z2. Then, the fused z value of the position can be expressed as: z = (z1 / n1+z2 / n2) / 2. If no valid point cloud is found at a certain position (i.e., n1 and n2 are both 0), it is marked as an invalid point, indicating that there is no available point cloud data at this position. For the position p without point cloud data in the query result, that is, when n1 and n2 are both zero, the current point p is defined as an invalid point. This step ensures that missing or unmatched data is correctly identified in the final fused point cloud. Loop through each coordinate position in the fusion memory and repeat the above calculation process until all points are processed. In this way, not only the effective fusion of point cloud data is achieved, but also the holes that may appear in the original scanning process are successfully filled, thereby improving the integrity and accuracy of the three-dimensional model. It is used to solve the problem of missing data that may be encountered by dual-line laser scanners in a single scan. By performing meticulous fusion processing on the point cloud transformed into the virtual space, it is ensured that the final generated three-dimensional model is both continuous and smooth, significantly reducing the hole problem in the original data.

[0141] Reference Figure 2 , is a structural block diagram of a dual-line laser point cloud hole filling device in one embodiment of the present application, the device includes:

[0142] An acquisition module 100 is used to acquire point cloud data of a first camera and a second camera, wherein the first camera and the second camera are installed horizontally opposite to each other with dual-line lasers;

[0143] Establishing module 200, for extracting feature point cloud and reference surface point cloud based on the point cloud data, and establishing a virtual coordinate system;

[0144] A calculation module 300, configured to calculate the transformation matrices of the first camera and the second camera to a virtual coordinate system, respectively, based on the feature point cloud and the reference plane point cloud;

[0145] A transformation module 400, configured to perform spatial transformation on the point clouds collected by the first camera and the second camera using the calculated transformation matrix;

[0146] The fusion module 500 is used to fuse the point cloud transformed into the virtual space and fill the empty point cloud. In one embodiment, the above device also includes an initialization module, and the initialization module includes:

[0147] A system initialization unit, used for initializing a machine system for controlling the first camera and the second camera;

[0148] The acquisition unit is used to determine the starting coordinates for starting acquisition according to the size of the calibration block to be scanned;

[0149] The setting unit is used to set the end coordinates of the machine according to the end position of the acquisition, so as to realize the alignment of the point cloud data acquired by the first camera and the second camera in space.

[0150] In one embodiment, the acquisition module 100 includes:

[0151] A sending unit, configured to send a trigger signal to the first camera and the second camera when recognizing that the machine has reached the starting coordinates;

[0152] A pulse sending unit, used to control the first camera and the second camera to start point cloud data acquisition based on a trigger signal, wherein during the operation of the machine, pulse signals are continuously sent to the cameras according to a set pulse interval to achieve synchronous acquisition of point cloud data;

[0153] An end sending unit, used to send an end signal to the first camera and the second camera to stop collecting point cloud data when the machine reaches a predetermined end coordinate;

[0154] A data acquisition unit is used to acquire point cloud data collected by the first camera and the second camera.

[0155] In one embodiment, the establishment module 200 includes:

[0156] an identification and extraction unit, used to identify and extract point clouds having the same features as the specific feature point cloud on the calibration block in the point cloud data collected by the first camera and the point cloud data collected by the second camera, to obtain the feature point cloud;

[0157] A first selection unit, configured to select a preset area from the point cloud data acquired by the first camera as a first reference plane point cloud;

[0158] A second selection unit is used to select a preset area as a second reference plane point cloud from the point cloud data acquired by the second camera; wherein the first reference plane point cloud and the second reference plane point are located in the same plane on the calibration block;

[0159] The coordinate establishing unit is used to establish a virtual coordinate system by using the extracted feature point cloud, the first reference plane point cloud and the second reference plane point cloud.

[0160] In one embodiment, the calculation module 300 includes:

[0161] An offset value calculation unit, used to calculate the feature point cloud extracted from the first camera and the feature point cloud extracted from the second camera, and determine the offset value of the first camera and the second camera in the xyz direction in the virtual coordinate system;

[0162] An extraction and calculation unit, used to perform point cloud plane fitting according to the reference surface point cloud extracted by the first camera and the reference surface point cloud extracted by the second camera, and calculate the position and rotation relationship of the first camera and the second camera relative to the virtual coordinate system;

[0163] The rotation calculation unit is used to obtain the transformation matrix of the first camera and the second camera in the virtual coordinate system according to the offset values ​​of the first camera and the second camera in the virtual coordinate system and the rotation relationship of the xyz coordinate axis.

[0164] In one embodiment, the offset value calculation unit includes:

[0165] A fitting unit, used to perform circle fitting on the feature point clouds extracted by the first camera and the second camera to obtain the circle center coordinates p1(x1, y1, z1) and p2(x2, y2, z2);

[0166] A filtering unit, configured to apply a statistical filtering algorithm to the reference plane point cloud extracted by the first camera and the second camera to remove noise points;

[0167] A solving unit is used to solve the plane equation parameters for the cleaned reference plane point cloud using the least square method to obtain the plane equation Ax+By+Cz+d=0, where C=1;

[0168] The z value calculation unit is used to calculate the z1 and z2 values ​​of the pair according to the plane equation and the coordinates of the circle center (x1, y1) and (x2, y2);

[0169] The offset value calculation subunit is used to calculate the offset values ​​of the two cameras in the x, y, and z directions, which are x1-x2, y1-y2, and z1-z2 respectively. In one embodiment, the above fusion module 500 includes:

[0170] A point cloud acquisition unit, used to acquire a transformed point cloud after the point cloud collected by the first camera and the second camera is transformed;

[0171] Establishing a unit for establishing the x-axis and y-axis resolutions of the virtual coordinate system using the point cloud resolutions of the two cameras;

[0172] A range determination unit, used to calculate the minimum and maximum x, y values ​​of the transformed point cloud and determine the spatial range of the point cloud;

[0173] A radius search unit, for each spatial position point p(x0, y0) in the virtual coordinate system, querying the point clouds of the first camera and the second camera based on the search radius, and calculating the average of the z values;

[0174] The invalid marking unit is used to mark the point p without point cloud data in the query result as an invalid point;

[0175] The traversal unit is used to traverse all points and complete the fusion and hole filling of point clouds.

[0176] Reference Figure 3 In an embodiment of the present application, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, an internal memory, a storage medium (non-volatile storage medium) and a network interface connected via a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes the above-mentioned storage medium (non-volatile storage medium) and the internal memory. The storage medium (non-volatile storage medium) stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the storage medium (non-volatile storage medium). The database of the computer device is used to store usage data in a dual-line laser point cloud hole filling method process, etc. The network interface of the computer device is used to communicate with an external terminal through a network connection. Furthermore, the above-mentioned computer device can also be provided with an input device and a display screen, etc. When the above computer program is executed by a processor, a dual-line laser point cloud hole filling method is implemented, which includes the following steps: obtaining point cloud data from a first camera and a second camera, wherein the first camera and the second camera are installed horizontally opposite to each other with dual-line lasers; extracting feature point clouds and reference point clouds based on the point cloud data, and establishing a virtual coordinate system; calculating the transformation matrices of the first camera and the second camera to the virtual coordinate system respectively according to the feature point clouds and the reference point clouds; using the calculated transformation matrix, performing spatial transformation on the point clouds collected by the first camera and the second camera; fusing the point clouds transformed to the virtual space to fill the hole point clouds. Those skilled in the art can understand that Figure 3The structure shown in is merely a block diagram of a portion of the structure related to the present application solution and does not constitute a limitation on the computer device to which the present application solution is applied.

[0177] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, a method for filling a hole in a two-line laser point cloud is implemented, including the following steps: obtaining point cloud data of a first camera and a second camera, wherein the first camera and the second camera are installed horizontally opposite to each other with two-line lasers; extracting feature point clouds and reference surface point clouds based on the point cloud data, and establishing a virtual coordinate system; calculating the transformation matrix of the first camera and the second camera to the virtual coordinate system respectively according to the feature point cloud and the reference surface point cloud; using the calculated transformation matrix, performing spatial transformation on the point clouds collected by the first camera and the second camera; fusing the point clouds transformed to the virtual space to fill the hole point clouds. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0178] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0179] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, device, article or method including the element.

[0180] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A dual-line laser point cloud hole filling method, characterized in that: The method comprises: Acquire point cloud data of a first camera and a second camera, wherein the first camera and the second camera are installed with dual-line lasers facing each other horizontally; Extracting feature point clouds and reference surface point clouds based on the point cloud data, and establishing a virtual coordinate system; Calculating transformation matrices of the first camera and the second camera to a virtual coordinate system respectively according to the feature point cloud and the reference plane point cloud; Using the calculated transformation matrix, spatially transform the point clouds collected by the first camera and the second camera; The point clouds transformed into the virtual space are fused to fill the void point clouds.

2. The dual-line laser point cloud hole filling method according to claim 1, characterized in that: Before the step of acquiring the point cloud data of the first camera and the second camera, the method includes: Initializing a machine system that controls the first camera and the second camera; Determine the starting coordinates for starting acquisition according to the size of the calibration block to be scanned; The end coordinates of the machine are set according to the end position of the acquisition to achieve the alignment of the point cloud data acquired by the first camera and the second camera in space.

3. The dual-line laser point cloud hole filling method according to claim 2, characterized in that: The step of acquiring point cloud data of the first camera and the second camera comprises: When it is recognized that the machine reaches the starting coordinates, a trigger signal is sent to the first camera and the second camera; Based on the trigger signal, the first camera and the second camera are controlled to start the point cloud data collection, wherein during the operation of the machine, pulse signals are continuously sent to the camera according to a set pulse interval to achieve synchronous collection of point cloud data; When the machine reaches the predetermined end coordinates, an end signal is sent to the first camera and the second camera to stop collecting point cloud data; Acquire point cloud data collected by the first camera and the second camera.

4. The dual-line laser point cloud hole filling method according to claim 1, characterized in that: The step of extracting a feature point cloud and a reference point cloud based on the point cloud data and establishing a virtual coordinate system comprises: Identify and extract point clouds having the same features as the specific feature point cloud on the calibration block from both the point cloud data collected by the first camera and the point cloud data collected by the second camera, and obtain the feature point cloud; Selecting a preset area from the point cloud data acquired by the first camera as a first reference surface point cloud; Selecting a preset area from the point cloud data acquired by the second camera as a second reference surface point cloud; wherein the first reference surface point cloud and the second reference surface point are located in the same plane on the calibration block; A virtual coordinate system is established using the extracted feature point cloud, the first reference plane point cloud, and the second reference plane point cloud.

5. The dual-line laser point cloud hole filling method according to claim 1, characterized in that: The step of respectively calculating the transformation matrices of the first camera and the second camera to the virtual coordinate system according to the feature point cloud and the reference plane point cloud comprises: Calculate the feature point cloud extracted from the first camera and the feature point cloud extracted from the second camera to determine the offset value of the first camera and the second camera in the xyz direction in the virtual coordinate system; Performing point cloud plane fitting based on the reference surface point cloud extracted by the first camera and the reference surface point cloud extracted by the second camera, and calculating the position and rotation relationship of the first camera and the second camera relative to the virtual coordinate system; According to the offset values ​​of the first camera and the second camera in the virtual coordinate system and the rotation relationship of the xyz coordinate axis, the transformation matrices of the first camera and the second camera in the virtual coordinate system are obtained.

6. The dual-line laser point cloud hole filling method according to claim 5, characterized in that: The step of calculating the feature point cloud extracted from the first camera and the feature point cloud extracted from the second camera to determine the offset value of the first camera and the second camera in the xyz direction in the virtual coordinate system includes: Perform circle fitting on the feature point clouds extracted by the first camera and the second camera to obtain the circle center coordinates p1(x1, y1, z1) and p2(x2, y2, z2); Applying a statistical filtering algorithm to the reference plane point cloud extracted by the first camera and the second camera to remove noise points; The least square method is used to solve the plane equation parameters of the cleaned reference surface point cloud, and the plane equation Ax+By+Cz+d=0 is obtained, where C=1; According to the plane equation and the coordinates of the center of the circle (x1, y1) and (x2, y2), calculate the z1 and z2 values ​​of the pair; Calculate the offset values ​​of the two cameras in the x, y, and z directions, which are x1-x2, y1-y2, and z1 respectively. - z2.

7. The dual-line laser point cloud hole filling method according to claim 1, characterized in that: The step of fusing the point cloud transformed into the virtual space and filling the hole point cloud comprises: Obtaining transformed point clouds after the point clouds collected by the first camera and the second camera are transformed; The point cloud resolutions of the two cameras establish the x-axis and y-axis resolutions of the virtual coordinate system; Calculate the minimum and maximum x and y values ​​of the transformed point cloud to determine the spatial range of the point cloud; For each spatial position point p(x0, y0) in the virtual coordinate system, query the point clouds of the first camera and the second camera based on the search radius and calculate the mean of the z value; For the point p without point cloud data in the query result, it is marked as an invalid point; Traverse all points to complete the fusion of point clouds and fill holes.

8. A dual-line laser point cloud hole filling device, characterized in that: include: An acquisition module, used for acquiring point cloud data of a first camera and a second camera, wherein the first camera and the second camera are installed with dual-line lasers facing each other horizontally; Establishing a module, for extracting a feature point cloud and a reference surface point cloud based on the point cloud data, and establishing a virtual coordinate system; A calculation module, used to calculate the transformation matrix of the first camera and the second camera to the virtual coordinate system according to the feature point cloud and the reference plane point cloud; A transformation module, used to perform spatial transformation on the point clouds collected by the first camera and the second camera using the calculated transformation matrix; The fusion module is used to fuse the point cloud transformed into the virtual space and fill the hole point cloud.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images