Application method of multi-sensor fusion technology in venue three-dimensional scanning

Abstract

The invention provides an application method of a multi-sensor fusion technology in venue three-dimensional scanning, and belongs to the technical field of three-dimensional scanning, and the method comprises the steps: carrying out the pre-scanning of a target venue based on a laser radar, obtaining initial three-dimensional contour data according to the output quality and unit attributes, and constructing a spatial topology model; a Kalman filtering dynamic compensation model is constructed based on a multi-sensor combination monitoring motion trail and in-museum light intensity variation; and performing point mapping compensation on the spatial topology model by depending on a shooting result of the same visible light camera, and constructing a spatial compensation vector of each model point in combination with the dynamic compensation matching relationship to adjust the corresponding spatial topology model, thereby obtaining and outputting an effective topology model. The precision, the integrity and the anti-interference capability of three-dimensional scanning of the venue are improved, and reliable three-dimensional model support is provided for related applications of the venue.

Description

Technical Field

[0001] This invention relates to the field of 3D scanning technology, and in particular to a method for applying multi-sensor fusion technology in 3D scanning of venues. Background Technology

[0002] With the widespread application of 3D scanning technology in the planning, design, operation, maintenance, renovation, and upgrading of large venues such as stadiums, convention centers, and theaters, higher demands are being placed on the accuracy, completeness, and dynamic adaptability of the scanned data. Existing 3D scanning technologies for venues largely rely on single sensors (such as LiDAR) for data acquisition, which has several limitations: First, LiDAR is easily affected by obstructions within the venue (such as temporary facilities and pedestrian traffic), leading to missing point cloud data. It is also sensitive to external factors such as ambient light and dust concentration, resulting in insufficient stability of the output data. Second, the movement trajectories of dynamic objects within the venue (such as pedestrians and mobile devices) can interfere with the scanning results. Existing technologies lack effective dynamic compensation mechanisms, making it difficult to eliminate modeling errors caused by dynamic interference. Third, the measurement dimensions of a single sensor are limited, making it impossible to comprehensively utilize multi-dimensional information to optimize the 3D model. This results in insufficient fit between the constructed spatial topology model and the actual venue, and the accuracy is insufficient to meet the needs of high-precision applications (such as venue equipment installation and calibration, and virtual simulation scene construction).

[0003] To address the aforementioned technical problems, this invention proposes a method for applying multi-sensor fusion technology in 3D scanning of venues. Summary of the Invention

[0004] This invention provides a method for applying multi-sensor fusion technology in venue 3D scanning. By working collaboratively with LiDAR and multiple types of sensors, combined with environmental adaptive acquisition, dynamic compensation modeling, and multi-source data fusion optimization, the accuracy, completeness, and anti-interference capability of venue 3D scanning are improved, providing reliable 3D model support for venue-related applications.

[0005] This invention provides a method for applying multi-sensor fusion technology in 3D scanning of venues, including:

[0006] Step 1: Pre-scan the target venue based on LiDAR, and obtain initial three-dimensional contour data and construct a spatial topology model based on the output quality determined by the output signal of LiDAR in each venue unit and the unit attributes of the corresponding venue unit determined by the venue construction drawing;

[0007] Step 2: Based on real-time monitoring of the motion trajectory of dynamic objects inside the target venue and the change in lighting intensity inside the venue using a combination of multiple sensors, construct a Kalman filter dynamic compensation model;

[0008] Step 3: Obtain the shooting results of each venue unit by cameras with different visible light, and perform point mapping compensation on the spatial topology model based on the shooting results of the same visible light camera. Combine the dynamic compensation model of Kalman filter with the dynamic compensation matching relationship of each model point to construct the spatial compensation vector of each model point.

[0009] Step 4: Adjust the corresponding spatial topology model based on the spatial compensation vector of each model point to obtain an effective topology model and output it.

[0010] Preferred options also include:

[0011] When pre-scanning the target venue using lidar, the environmental perception module collects the light intensity, dust concentration and air turbulence coefficient in the venue in real time, and identifies suspected occlusion areas from the point cloud data output by lidar in real time to obtain the current occlusion distribution.

[0012] The current occlusion distribution is divided into occlusion degree categories, and the classification results are firstly labeled according to the scanning order. At the same time, the total area of ​​the same occlusion degree is determined, and a second label is performed according to the concentration location of each total area and the scanning angle of the center point of the concentration location during the scanning process, and combined with the minimum and maximum angles based on the scanning order for each location point in the concentration location.

[0013] Based on the first and second order calibrations, and combined with the distribution uniformity, the re-scanning process is determined, and the transmission power during re-scanning is n0 times that of conventional scanning, so as to realize environmental adaptation and occlusion compensation acquisition of the pre-scanned point cloud.

[0014] Preferably, acquiring initial 3D contour data and constructing a spatial topology model includes:

[0015] The pre-scanning result is regarded as the first three-dimensional contour data, and the first three-dimensional contour data is divided into several sub-data according to the smallest contour unit. The basic contour is drawn for each sub-data to obtain the sub-contour.

[0016] A two-dimensional matrix is ​​constructed based on the output quality determined by the output signal of the lidar in each venue unit and the unit attributes of the corresponding venue unit determined by the venue construction plan;

[0017] Based on the location comparison relationship, determine all the sub-contours involved in each venue unit, and construct the offset matrix of the corresponding venue unit according to the offset coefficient between the corresponding venue unit and each corresponding sub-contour;

[0018] The offset matrix is ​​time-aligned with the set of scanning angle changes of the lidar in the corresponding venue unit, and coupled with each sub-contour involved in the corresponding venue unit to obtain the initial three-dimensional contour data.

[0019] Determine the structure to which each sub-contour belongs and its corresponding two-dimensional matrix in the venue construction drawing, and determine the construction accuracy of the corresponding sub-contour;

[0020] According to the construction accuracy, the unit spatial modeling shape based on each sub-contour is obtained from the accuracy-design lookup table, and a spatial topology model is constructed based on the initial three-dimensional contour data.

[0021] Preferably, the multi-sensor combination includes: an infrared thermal imaging sensor, a millimeter-wave sensor, a vision sensor, and a light sensor, wherein the infrared thermal imaging sensor and the millimeter-wave sensor respectively generate the first motion trajectory of the first object and the second motion trajectory of the second object, the vision sensor collects the contour features of the dynamic object to distinguish the object type, and the light sensor is arranged in a grid on the top of the venue to collect the absolute value of the light intensity and the rate of change per second in the grid unit in real time.

[0022] Based on the similarity between the first and second motion trajectories, the infrared thermal imaging sensor and the millimeter-wave sensor are correlated and matched to monitor targets. Simultaneously, for different types of dynamic objects, differentiated motion feature baselines are preset and based on… Obtain the motion trajectory coordination coefficients of each dynamic object. ,in, This is a preset synergy coefficient correction factor; The preset deviation threshold; This represents the deviation between the real-time motion parameters and the characteristic baseline.

[0023] In monitoring changes in indoor lighting intensity, the attenuation rate of lighting intensity within the object's projection area is calculated by combining the real-time position coordinates of dynamic objects, and based on... Correction factor for generating changes in light intensity ,in, The preset illumination correction factor, The light intensity attenuation rate; The preset attenuation rate threshold;

[0024] Based on the motion trajectory coordination coefficient With correction factor Adjust the noise covariance of the model filtering process;

[0025] The dynamic object contour features acquired by the visual sensor are used as observation constraints in the filtering model, and a Kalman filter dynamic compensation model is constructed by combining the adjusted covariance.

[0026] Preferably, step 3 includes:

[0027] A dynamic weight model of image-topology unit is constructed, and the weight is positively correlated with the unit structure complexity of the spatial topology model and the spectral responsiveness of building materials. In addition, dynamic image coefficients are assigned to the model points in the region traversed by the dynamic object in the Kalman filter dynamic compensation model.

[0028] When performing point mapping compensation on the spatial topology model based on the shooting results of the same visible light camera, a sub-pixel corner detection algorithm is used to extract image feature points, and the external parameter matrix of the camera and the lidar is obtained based on multi-view geometric calibration, and the image feature points are mapped to the three-dimensional coordinate system of the spatial topology model.

[0029] Construct the initial compensation vector for each model point under the same visible light camera. ;

[0030] Iterate through the initial compensation vectors of each model point under different visible light cameras. The compensation factors are determined, and a factor set is constructed to analyze the undetermined coefficients of the corresponding model points;

[0031] When the undetermined coefficient is less than r0, the corresponding model point is determined as the point to be optimized;

[0032] Otherwise, draw the unit frame to the corresponding model point, and determine the undetermined coefficients of the remaining points in the unit frame other than the corresponding model point, and determine the first adjustment factor based on the points in the unit frame that are less than r0.

[0033] Draw a minimum edge bounding plot for all points to be optimized, and determine the second adjustment factor;

[0034] Based on the first adjustment factor and the second adjustment factor, for all model points After comprehensive processing, the spatial compensation vectors of the corresponding model points are obtained.

[0035] Preferably, an initial compensation vector is constructed for each model point under the same visible light camera. ,include:

[0036] ,in, The image feature points under the same visible light from the camera are represented by sub-pixel-level three-dimensional coordinate vectors. The three-dimensional coordinate vector of the original point in the spatial topology model. Let be the residual vector of the Kalman filter dynamic compensation model. Image mapping weights, This refers to the dynamic compensation weight adjusted based on the dynamic response coefficient.

[0037] Preferably, the corresponding spatial topology model is adjusted based on the spatial compensation vector of each model point, including:

[0038] Fine-grained feature extraction is performed on the spatial compensation vectors of all model points, using the following formula:

[0039] ,in, This is the default feature representation for spatial compensation. For multi-scale interactive attention modules, i.e., through parameters Adapt to compensation vector features of different scales; For adaptive residual structure, i.e. through coefficients Dynamically adjust residual weights;

[0040] Based on the extracted features, all model points are divided into clusters according to accuracy requirements, and a scene-cluster dynamic priority mapping table is constructed in combination with the venue application scenario.

[0041] Each precision requirement cluster is treated as an independent adjustment unit. High precision requirement clusters are adjusted using point cloud-topology residual dual verification, medium precision requirement clusters are adjusted using point cloud residual single verification, and low precision requirement clusters are adjusted using simplified verification. At the same time, different precision requirement clusters in the corresponding application scenarios are adjusted according to the scenario-cluster dynamic priority mapping table.

[0042] Redundant nodes are removed from the adjusted topology model to obtain an effective topology model.

[0043] Preferably, the minimum edge bounding is drawn for all points to be optimized, and the second adjustment factor is determined, including:

[0044] The ratio of the number of points to be optimized within the enclosed area to the area of ​​the region is determined to obtain the point density index;

[0045] Quantify the point uniformity of the point to be optimized within the enclosing region;

[0046] Analyze the spatial overlap between the surrounding area and different functional areas of the venue to obtain functional correlation indicators;

[0047] Based on the point density index, point uniformity, and functional correlation index, a second adjustment factor is obtained.

[0048] Compared with the prior art, the beneficial effects of this application are as follows:

[0049] By combining lidar pre-scanning with multi-sensor fusion monitoring, a closed-loop process for 3D scanning of the venue is achieved, from initial modeling to dynamic compensation and model optimization. This effectively solves problems such as occlusion interference, environmental sensitivity, and dynamic errors associated with single-sensor scanning. The constructed effective topology model has high accuracy and good integrity, and can accurately reflect the actual spatial structure and dynamic changes of the environment of the venue, providing reliable 3D data support for the planning, operation, maintenance, and renovation of the venue.

[0050] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0051] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0052] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0053] Figure 1 This is a flowchart illustrating a method for applying multi-sensor fusion technology in 3D scanning of a venue, as described in an embodiment of the present invention. Detailed Implementation

[0054] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0055] This invention provides a method for applying multi-sensor fusion technology in 3D scanning of venues, such as... Figure 1 As shown, it includes:

[0056] Step 1: Pre-scan the target venue based on LiDAR, and obtain initial three-dimensional contour data and construct a spatial topology model based on the output quality determined by the output signal of LiDAR in each venue unit and the unit attributes of the corresponding venue unit determined by the venue construction drawing;

[0057] Step 2: Based on real-time monitoring of the motion trajectory of dynamic objects inside the target venue and the change in lighting intensity inside the venue using a combination of multiple sensors, construct a Kalman filter dynamic compensation model;

[0058] Step 3: Obtain the shooting results of each venue unit by cameras with different visible light, and perform point mapping compensation on the spatial topology model based on the shooting results of the same visible light camera. Combine the dynamic compensation model of Kalman filter with the dynamic compensation matching relationship of each model point to construct the spatial compensation vector of each model point.

[0059] Step 4: Adjust the corresponding spatial topology model based on the spatial compensation vector of each model point to obtain an effective topology model and output it.

[0060] In this embodiment, a lidar is a sensor that measures the distance, orientation, and height of a target by emitting a laser beam. A lidar with a detection range of 0.1-100m and an angular resolution of 0.1° can be selected, which is suitable for large-scale rapid scanning of venues.

[0061] In this embodiment, the target venue refers to various large enclosed or semi-enclosed spaces that require 3D scanning.

[0062] In this embodiment, the venue unit is the smallest functional or spatial unit that divides the target venue according to preset rules. For example, it can be divided into four types of units according to function: audience area, exhibition area, passage area, and equipment area. Each unit is 10m×10m in size.

[0063] In this embodiment, output quality is a reliability indicator of the data output by the LiDAR when scanning each venue unit. It is comprehensively evaluated by three parameters: point cloud density, data noise value, and effective point ratio. Among them, point cloud density... Noise value Percentage of effective points When the output quality level is excellent, the output quality level is good if one or two indicators do not meet the corresponding standards; otherwise, the output quality level is poor.

[0064] In this embodiment, the venue construction drawing is the design and construction drawing of the target venue, which includes information such as the venue's floor plan, structural dimensions, building materials, and functional zoning. It adopts CAD format construction drawings with a scale of 1:100.

[0065] In this embodiment, the unit attributes are the inherent characteristic information of the venue unit, including unit function, building materials, structural complexity, etc.

[0066] In this embodiment, the initial three-dimensional contour data is a set of three-dimensional data that reflects the approximate shape and spatial structure of the venue after pre-scanning by lidar and preliminary processing.

[0067] In this embodiment, the spatial topology model is constructed based on the initial three-dimensional contour data and is a three-dimensional model that can describe the positional relationship, connection method and geometric shape of each spatial unit in the venue.

[0068] In this embodiment, a multi-sensor combination is used to collaboratively monitor dynamic objects and light intensity within the venue, including an infrared thermal imaging sensor, a millimeter-wave sensor, a visual sensor, and a light sensor.

[0069] In this embodiment, dynamic objects are objects in motion within the venue, including pedestrians, mobile devices, temporary mobile facilities, etc., mainly targeting pedestrians (1.5-1.9m tall) and mobile display cases.

[0070] In this embodiment, the motion trajectory is the path of the dynamic object's position change within a unit of time, represented by a three-dimensional coordinate sequence (x,t,y,t,z,t) with a time interval of 0.1s. For example, an infrared thermal imaging sensor generates a first motion trajectory by capturing the thermal radiation signal of the dynamic object, such as the trajectory sequence of pedestrian A: (10,0.1,20,0.1,1.7,0.1), (10.2,0.2,20.1,0.2,1.7,0.2), etc. A millimeter-wave sensor generates a second motion trajectory by emitting millimeter waves and receiving reflected signals, such as the trajectory sequence of pedestrian A: (10.1,0.1,20.05,0.1,1.68,0.1), (10.22,0.2,20.12,0.2,1.68,0.2), etc.

[0071] In this embodiment, the change in light intensity is the change in the light intensity inside the venue over time, measured in lux, and the change is equal to the current intensity minus the initial intensity.

[0072] In this embodiment, the Kalman filter dynamic compensation model is a mathematical model built based on the Kalman filter algorithm to compensate for modeling errors caused by dynamic object interference and changes in light intensity. The core is to optimize the model accuracy through state estimation and error correction. The definition of dynamic compensation matching relationship refers to the mapping correspondence between the residual vector output by the Kalman filter dynamic compensation model and the coordinates of the model points.

[0073] In this embodiment, point mapping compensation is the process of mapping the feature points of the two-dimensional image captured by the camera to the three-dimensional coordinate system of the spatial topology model through coordinate transformation, thereby correcting the position of the corresponding points on the model.

[0074] In this embodiment, the spatial compensation vector is used to adjust the three-dimensional vector of the position of each model point in the spatial topology model. The calculation is obtained through multi-source data fusion, which can eliminate the positional deviation of model points. For example, the point mapping compensation deviation of model point P(20,30,5) is (0.002,0.001,0.003), and the dynamic compensation deviation is (0.001,0.002,0.001). The weighting coefficients are 0.6 and 0.4, respectively. Then the spatial compensation vector is (0.002×0.6+0.001×0.4,0.001×0.6+0.002×0.4,0.003×0.6+0.001×0.4)=(0.0016,0.0014,0.0022).

[0075] The beneficial effects of the above technical solution are as follows: by combining lidar pre-scanning with multi-sensor fusion monitoring, a closed-loop process of 3D scanning of the venue is realized from initial modeling to dynamic compensation and model optimization. This effectively solves the problems of occlusion interference, environmental sensitivity, and dynamic errors caused by single sensor scanning. The constructed effective topology model has high accuracy and good integrity, and can accurately reflect the actual spatial structure and dynamic changes of the environment of the venue, providing reliable 3D data support for the planning, operation and maintenance, and renovation of the venue.

[0076] This invention provides a method for applying multi-sensor fusion technology in 3D scanning of venues, and further includes:

[0077] When pre-scanning the target venue using lidar, the environmental perception module collects the light intensity, dust concentration and air turbulence coefficient in the venue in real time, and identifies suspected occlusion areas from the point cloud data output by lidar in real time to obtain the current occlusion distribution.

[0078] The current occlusion distribution is divided into occlusion degree categories, and the classification results are firstly labeled according to the scanning order. At the same time, the total area of ​​the same occlusion degree is determined, and a second label is performed according to the concentration location of each total area and the scanning angle of the center point of the concentration location during the scanning process, and combined with the minimum and maximum angles based on the scanning order for each location point in the concentration location.

[0079] Based on the first and second order calibrations, and combined with the distribution uniformity, the re-scanning process is determined, and the transmission power during re-scanning is n0 times that of conventional scanning, so as to realize environmental adaptation and occlusion compensation acquisition of the pre-scanned point cloud.

[0080] In this embodiment, the environmental sensing module consists of a light sensor, a dust concentration sensor, and an air turbulence coefficient sensor, and is used to collect environmental parameters in the venue in real time. The light sensor is selected from TSL2561, the dust concentration sensor is selected from PMS5003, and the air turbulence coefficient sensor is selected from FT700.

[0081] In this embodiment, the suspected occlusion area is the area where point cloud data is missing or the percentage of valid points is lower than a preset threshold (80%) when the lidar is scanned, such as behind exhibits temporarily piled up in the venue or in areas with dense crowds.

[0082] In this embodiment, the current occlusion distribution is the set of the location, range and shape of all suspected occlusion areas in the venue at a certain moment, represented by a two-dimensional coordinate range (x1, y1, x2, y2), such as the exhibit occlusion area (30, 40, 35, 45).

[0083] In this embodiment, the degree of occlusion is divided into three levels: mild, moderate, and severe, based on the percentage of effective points in the occluded area and the point cloud missing rate. A percentage of effective points of 70%-80% is considered mild occlusion, 50%-70% is considered moderate occlusion, and less than 50% is considered severe occlusion.

[0084] In this embodiment, the first order calibration is to sort and calibrate the areas with different degrees of occlusion according to the scanning order of the lidar (starting from the northeast corner of the venue clockwise). Assuming that the scanning order first encounters area A (mild), area B (moderate), area C (severe), and area D (mild), the calibration result is mild occlusion Q1-1 (A) and Q1-2 (D), moderate occlusion Q2-1 (B), and severe occlusion Q3-1 (C).

[0085] In this embodiment, the total area is the sum of the areas of all occluded areas under the same degree of occlusion. For example, there are three areas with slight occlusion, with areas of... The total area is .

[0086] In this embodiment, the central location is the geometric center of all occlusion areas under the same degree of occlusion.

[0087] In this embodiment, the scanning angle is the angle from which the laser beam emitted by the lidar reaches the center point of the obstructed area, including the horizontal angle and the vertical angle. For example, the horizontal scanning angle of the center point (25,30) is 45° and the vertical scanning angle is 0°.

[0088] The minimum angle and maximum angle are the minimum and maximum values ​​of the scanning angle of the center point of all occluded areas under the same degree of occlusion in the scanning sequence. For example, the horizontal scanning angle range of a lightly occluded area is 30°-60°, with a minimum angle of 30° and a maximum angle of 60°.

[0089] The second order calibration is based on the total area of ​​the division with the same degree of occlusion, the scanning angle and angle range of the concentrated position, and a secondary sorting and calibration of the occluded areas. The larger the total area of ​​division and the closer the scanning angle is to the center angle (180°), the higher the sorting.

[0090] Calculate the total area divided for the same degree of occlusion, such as the lightly occluded area A ( ), D ( ), total area ;

[0091] Determine the concentration points (geometric centers): the center of A is (32.5, 42.5), the center of D is (40, 35), and the concentration point of slight shading is ((32.5+40) / 2, (42.5+35) / 2) = (36.25, 38.75);

[0092] Record the scanning angles of the concentrated positions, such as a horizontal angle of 55° and a vertical angle of 0°;

[0093] Determine the angle range: A's horizontal scanning angle is 45°-50°, D's horizontal scanning angle is 55°-60°, with a minimum angle of 45° and a maximum angle of 60°.

[0094] Sort by the total area percentage (A accounts for 57.1%, D accounts for 42.9%), with A ranked first. The final second order is marked as Q1-1-1 (A) and Q1-1-2 (D).

[0095] The rescanning process was determined by combining the first and second calibration results and the uniformity of the distribution of the obstructed area (A and D are dispersed). The rescanning sequence was set as Q1-1-1 (A) → Q1-1-2 (D) → Q2-1 (B) → Q3-1 (C). The rescanning range was 0.5m outward from the coordinate range of the obstructed area (e.g., the rescanning range of A is (29.5, 39.5, 35.5, 45.5)). The rescanning parameters were set as follows: scanning frequency 15Hz, transmission power 15W (1.5 times the conventional power of 10W).

[0096] In this embodiment, the environment-adaptive acquisition method is a method in which the lidar automatically adjusts the scanning parameters (such as power and frequency) based on the environmental parameters (light, dust, turbulence) collected by the environmental perception module to adapt to changes in the environment.

[0097] In this embodiment, occlusion compensation acquisition is a method of acquiring complete data of occluded areas by performing supplementary scanning of occluded areas through a supplementary scanning process to make up for the data loss of the initial scan.

[0098] The beneficial effects of the above technical solution are as follows: by monitoring the venue's environmental parameters in real time through the environmental perception module, accurately identifying occlusion areas and classifying the degree of occlusion, formulating a scientific rescanning process through dual-sequence calibration, and combining transmission power adjustment to achieve environmental adaptive acquisition and occlusion compensation acquisition of the pre-scanned point cloud, the solution effectively solves the problem of data loss caused by environmental interference and occlusion in traditional scanning, significantly improves the integrity and reliability of the pre-scanned point cloud data, and lays a high-quality data foundation for subsequent 3D model construction.

[0099] This invention provides a method for applying multi-sensor fusion technology in 3D scanning of venues, acquiring initial 3D contour data and constructing a spatial topology model, including:

[0100] The pre-scanning result is regarded as the first three-dimensional contour data, and the first three-dimensional contour data is divided into several sub-data according to the smallest contour unit. The basic contour is drawn for each sub-data to obtain the sub-contour.

[0101] A two-dimensional matrix is ​​constructed based on the output quality determined by the output signal of the lidar in each venue unit and the unit attributes of the corresponding venue unit determined by the venue construction plan;

[0102] Based on the location comparison relationship, determine all the sub-contours involved in each venue unit, and construct the offset matrix of the corresponding venue unit according to the offset coefficient between the corresponding venue unit and each corresponding sub-contour;

[0103] The offset matrix is ​​time-aligned with the set of scanning angle changes of the lidar in the corresponding venue unit, and coupled with each sub-contour involved in the corresponding venue unit to obtain the initial three-dimensional contour data.

[0104] Determine the structure to which each sub-contour belongs and its corresponding two-dimensional matrix in the venue construction drawing, and determine the construction accuracy of the corresponding sub-contour;

[0105] According to the construction accuracy, the unit spatial modeling shape based on each sub-contour is obtained from the accuracy-design lookup table, and a spatial topology model is constructed based on the initial three-dimensional contour data.

[0106] In this embodiment, the first three-dimensional contour data is the venue's three-dimensional point cloud data obtained after pre-scanning by lidar and preliminary screening (removing obvious noise points and invalid points), which is the basis for constructing the initial three-dimensional contour data.

[0107] In this embodiment, the smallest contour unit is the smallest three-dimensional data unit that divides the first three-dimensional contour data according to a preset size, such as a cube unit with a size of 1m×1m×1m, that is, a cube unit with a size of 1m along each of the x, y, and z axes.

[0108] In this embodiment, sub-data is a subset of 3D point cloud data corresponding to the smallest contour unit. Each sub-data contains all valid points within the unit. For example, the point cloud data within a certain smallest contour unit (10-11m, 20-21m, 3-4m) is the sub-data of that unit.

[0109] In this embodiment, the sub-contour is a basic contour drawn based on sub-data that can reflect the spatial shape within the smallest contour unit. It is obtained by using a contour extraction algorithm (such as Canny edge detection + contour fitting), and its shape is a polygon or polyhedron.

[0110] In this embodiment, the two-dimensional matrix construction involves extracting the output quality parameters (point cloud density, noise value, effective point ratio) and unit attribute parameters (functional type, material type, structural complexity) of each venue unit (10m×10m), encoding them, and then constructing the two-dimensional matrix. For example, the output quality parameters of a certain venue unit in the exhibition area are ( If the element attribute parameter is (2, glass + concrete, 3), then the data in that row of the two-dimensional matrix is ​​(0.008m, 92%). .

[0111] In this embodiment, the positional correspondence is the spatial positional correspondence between the venue unit and the minimum outline unit. That is, each venue unit (10m×10m) contains 100 minimum outline units (1m×1m), and they are associated by coordinate range, such as the venue unit (0-10m, 0-10m) containing the minimum outline units (0-1m, 0-1m) to (9-10m, 9-10m).

[0112] In this embodiment, the offset coefficient is the deviation coefficient between the actual physical location and the designed location of the venue unit. It is calculated by comparing the design coordinates on the venue construction drawing and the actual coordinates scanned by the lidar. .

[0113] In this embodiment, the offset matrix is ​​constructed with the sub-contours within the venue unit as rows and the offset coefficients in the x, y, and z directions as columns, with a dimension of K×3 (K is the number of sub-contours within the venue unit). For example, if a venue unit contains 100 sub-contours, the offset matrix is ​​100×3, and the elements are the offset coefficients of each sub-contour in the x, y, and z directions.

[0114] In this embodiment, the set of scanning angle changes is the sequence of changes in the horizontal and vertical scanning angles during the process of LiDAR scanning a venue unit. For example, the horizontal angle changes from 30° to 60° and the vertical angle changes from -5° to 5°. The set of changes is {(30°, -5°), (31°, -4°), ..., (60°, 5°)}.

[0115] In this embodiment, the time alignment process is a method of matching and aligning the offset matrix with the set of changes in scanning angle according to the timestamp, so as to ensure that the offset coefficient and the scanning angle correspond at the same time point.

[0116] In this embodiment, the coupling process involves fusing the time-aligned offset matrix with the sub-contour data. A weighted summation algorithm is used to incorporate the offset coefficients into the sub-contour's coordinate data, correcting the sub-contour's positional deviation. ,in, The original coordinates of the sub-contour; These are the coordinates of the coupled sub-contour.

[0117] In this embodiment, the structure refers to the building structure type corresponding to the sub-outline in the venue construction drawing, such as load-bearing wall structure, beam structure, ground structure, ceiling structure, etc.

[0118] In this embodiment, the construction accuracy is the 3D modeling accuracy requirement corresponding to the sub-contour, which is determined by the importance and usage requirements of the structure. For example, the construction accuracy requirement for a load-bearing wall structure is ±0.005m, and for a ground structure it is ±0.01m.

[0119] In this embodiment, the accuracy-design lookup table is pre-defined, recording the design parameters such as unit space modeling shape and mesh density corresponding to different construction accuracies, as shown in Table 1:

[0120] Table 1 Accuracy-Design Comparison Table

[0121]

[0122] In this embodiment, the unit space modeling shape is selected from the accuracy-design lookup table based on the construction accuracy, and is used to construct the basic geometric shape of the sub-contour 3D model, such as a regular tetrahedron, a regular hexahedron, etc.

[0123] The beneficial effects of the above technical solution are as follows: through a series of steps such as minimum contour unit segmentation, two-dimensional matrix modeling, offset matrix coupling, and accuracy adaptation modeling, the process of acquiring initial three-dimensional contour data is refined, the construction logic of spatial topology model is clarified, the impact of venue unit attribute differences and scanning deviations on modeling accuracy is effectively reduced, and the constructed spatial topology model can accurately match the actual structure of the venue, providing a high-precision basic model for subsequent model optimization.

[0124] This invention provides a method for applying multi-sensor fusion technology in 3D scanning of venues. The multi-sensor combination includes an infrared thermal imaging sensor, a millimeter-wave sensor, a vision sensor, and a light sensor. The infrared thermal imaging sensor and the millimeter-wave sensor generate a first motion trajectory of a first object and a second motion trajectory of a second object, respectively. The vision sensor collects the contour features of dynamic objects to distinguish object types. The light sensor is arranged in a grid on the top of the venue to collect the absolute value of light intensity and the rate of change per second within the grid unit in real time.

[0125] Based on the similarity between the first and second motion trajectories, the infrared thermal imaging sensor and the millimeter-wave sensor are correlated and matched to monitor targets. Simultaneously, for different types of dynamic objects, differentiated motion feature baselines are preset and based on… Obtain the motion trajectory coordination coefficients of each dynamic object. ,in, This is a preset synergy coefficient correction factor; The preset deviation threshold; This represents the deviation between the real-time motion parameters and the characteristic baseline.

[0126] In monitoring changes in indoor lighting intensity, the attenuation rate of lighting intensity within the object's projection area is calculated by combining the real-time position coordinates of dynamic objects, and based on... Correction factor for generating changes in light intensity ,in, The preset illumination correction factor, The light intensity attenuation rate; The preset attenuation rate threshold;

[0127] Based on the motion trajectory coordination coefficient With correction factor Adjust the noise covariance of the model filtering process;

[0128] The dynamic object contour features acquired by the visual sensor are used as observation constraints in the filtering model, and a Kalman filter dynamic compensation model is constructed by combining the adjusted covariance.

[0129] In this embodiment, the first object / second object is a dynamic object monitored by the infrared thermal imaging sensor and the millimeter-wave sensor respectively. The two objects may be the same object or different objects, and both point to pedestrians or mobile devices in the venue.

[0130] In this embodiment, the outline features are the edge shape, size ratio, texture features, etc. of the dynamic object in the visual image. For example, the outline feature of a pedestrian is an upright rectangle with a width-to-height ratio of about 1:3; the outline feature of a mobile display cabinet is a cuboid with a width-to-height ratio of about 2:1.

[0131] In this embodiment, the object type is a dynamic object category divided according to contour features, which is divided into three categories: pedestrians, mobile display cabinets, and cleaning equipment.

[0132] In this embodiment, the grid layout involves evenly arranging the light sensors on the top of the venue according to a preset grid (such as 5m×5m) to ensure that there is one light sensor in each grid unit, thus fully covering the venue.

[0133] In this embodiment, the absolute value of light intensity is the actual light intensity within a grid cell at a certain moment, collected by the light sensor, and the unit is lux. For example, the absolute value of light intensity in a certain grid cell is 500 lux.

[0134] In this embodiment, the rate of change per second is the magnitude of change in light intensity within 1 second. The calculation formula is: (current intensity - intensity of the previous second) / 1s, with the unit being lux / s. For example, a rate of change of -5 lux / s means that the light intensity decreases by 5 lux per second.

[0135] In this embodiment, trajectory similarity is the degree of overlap between the first motion trajectory and the second motion trajectory. It is calculated using the Dynamic Time Warping (DTW) algorithm, and the similarity value ranges from 0 to 1. The closer it is to 1, the more consistent the trajectories are.

[0136] In this embodiment, the association matching is a process of determining whether the first object and the second object are the same dynamic object based on trajectory similarity. The similarity threshold is set to 0.85. When the similarity is ≥0.85, they are determined to be the same object, thus realizing the association of the sensor monitoring target.

[0137] In this embodiment, the motion characteristic baseline is a preset standard value for motion parameters for different types of dynamic objects, including velocity range, acceleration range, and frequency of change in motion direction. For example, the velocity baseline for a pedestrian is 0.5-1.5 m / s, and the acceleration baseline is -0.5-0.5 m / s. .

[0138] In this embodiment, the motion trajectory coordination coefficient A coefficient reflecting the degree of coordination between the real-time motion trajectory of a dynamic object and the feature baseline. The value ranges from 0 to 1. The closer it is to 1, the better the coordination. It is used to adjust the noise covariance of the filtering model.

[0139] In this embodiment, the coordination coefficient correction factor It is a preset constant used to adjust the calculation results of the motion trajectory coordination coefficient. Based on experimental data, it is calibrated and has a value of 0.95.

[0140] In this embodiment, a preset deviation threshold is used. It is the maximum permissible deviation between the preset real-time motion parameters of the dynamic object and the feature baseline, such as (Speed ​​deviation) (acceleration deviation).

[0141] In this embodiment, the deviation value between the real-time motion parameters and the characteristic baseline It is the difference between the real-time motion parameters of a dynamic object (such as actual velocity and actual acceleration) and the characteristic baseline, and the formula is: .

[0142] In this embodiment, the light intensity attenuation rate It is the percentage reduction in light intensity within the projection area caused by dynamic object occlusion, calculated using the following formula: And the value ranges from 0 to 0.3.

[0143] Correction factor for changes in light intensity The coefficient used to correct the measurement error of changes in light intensity has a value range of 0.8-1.2.

[0144] Illumination correction factor It is a preset constant used to adjust the calculation results of the correction factor, and =1.05.

[0145] Preset attenuation threshold It is the maximum allowable value of the preset light intensity attenuation rate, and =0.25.

[0146] The noise covariance is a matrix in the Kalman filter model that describes the statistical characteristics of observed noise and process noise. It is used to measure the degree of influence of noise on the filtering result. The larger the value, the greater the influence of noise.

[0147] The observation constraint term is used to limit the range of observation results of the Kalman filter model. It uses the contour features of the dynamic object collected by the visual sensor as a constraint to ensure that the filtering result conforms to the actual shape of the dynamic object.

[0148] In this embodiment, the Kalman filter dynamic compensation model is constructed by using the dynamic object contour features acquired by the visual sensor as observation constraints (such as the contour size range of pedestrians being 1.5-1.9m), combined with the adjusted noise covariance matrix Q, to construct the state equation of the Kalman filter. and observation equations Where A is the state transition matrix and B is the control matrix. To control the input, H represents process noise, and H represents the observation matrix. To observe the noise, a dynamic compensation model for the Kalman filter was finally obtained.

[0149] The beneficial effects of the above technical solution are as follows: through the coordinated deployment of multiple types of sensors, comprehensive monitoring of the dynamic object movement trajectory and venue lighting intensity changes can be achieved. Through trajectory association matching, coordination coefficient calculation, noise covariance adjustment and other means, the constructed Kalman filter dynamic compensation model can accurately quantify the modeling error caused by dynamic interference and illumination changes, providing a scientific dynamic compensation basis for subsequent spatial compensation of model points, and significantly improving the anti-interference ability and accuracy stability of the three-dimensional model.

[0150] This invention provides a method for applying multi-sensor fusion technology in 3D scanning of venues, step 3 including:

[0151] A dynamic weight model of image-topology unit is constructed, and the weight is positively correlated with the unit structure complexity of the spatial topology model and the spectral responsiveness of building materials. In addition, dynamic image coefficients are assigned to the model points in the region traversed by the dynamic object in the Kalman filter dynamic compensation model.

[0152] When performing point mapping compensation on the spatial topology model based on the shooting results of the same visible light camera, a sub-pixel corner detection algorithm is used to extract image feature points, and the external parameter matrix of the camera and the lidar is obtained based on multi-view geometric calibration, and the image feature points are mapped to the three-dimensional coordinate system of the spatial topology model.

[0153] Construct the initial compensation vector for each model point under the same visible light camera. ;

[0154] Iterate through the initial compensation vectors of each model point under different visible light cameras. The compensation factors are determined, and a factor set is constructed to analyze the undetermined coefficients of the corresponding model points;

[0155] When the undetermined coefficient is less than r0, the corresponding model point is determined as the point to be optimized;

[0156] Otherwise, draw the unit frame to the corresponding model point, and determine the undetermined coefficients of the remaining points in the unit frame other than the corresponding model point, and determine the first adjustment factor based on the points in the unit frame that are less than r0.

[0157] Draw a minimum edge bounding plot for all points to be optimized, and determine the second adjustment factor;

[0158] Based on the first adjustment factor and the second adjustment factor, for all model points After comprehensive processing, the spatial compensation vectors of the corresponding model points are obtained.

[0159] In this embodiment, the image-topology unit dynamic weight model is a mathematical model that assigns dynamic weights to each model point based on the unit structural features and dynamic object influence of the spatial topology model. The weight values ​​range from 0 to 1 and are used for weighted summation in subsequent compensation vector calculations. The weight calculation formula of the dynamic weight model is w = (structural complexity / 5 + spectral response) / 2. For example, the weight of this unit is w = (4 / 5 + 0.8) / 2 = (0.8 + 0.8) / 2 = 0.8. Combining the historical trajectory of dynamic objects in the Kalman filter dynamic compensation model, the number of times each model point is traversed by dynamic objects and the dwell time are counted. Dynamic image coefficients (1.0-1.5) are assigned to model points with high traversal frequency. For example, if a model point in the channel area is traversed 10 times, the dynamic image coefficient is 1.2. Finally, the comprehensive weight of this model point is 0.8 × 1.2 = 0.96.

[0160] Element structural complexity is an index describing the geometric complexity of a unit in a spatial topological model. It is comprehensively evaluated by the unit's surface curvature, number of edges and corners, and contour irregularity, with a value ranging from 1 to 5. The larger the value, the higher the complexity. For example, the complexity of a simple planar unit is 1, while the complexity of an irregular decorative unit is 5.

[0161] The spectral response of building materials is the degree to which the reflection and absorption characteristics of building materials match those of visible light. It is calculated by the degree of overlap between the spectral reflectance of the material and the spectrum captured by the camera. The value ranges from 0 to 1. For example, the spectral response of white concrete is 0.9, and the spectral response of ferrous metals is 0.3.

[0162] The dynamic image coefficient is an additional weighting coefficient assigned to model points in areas traversed by dynamic objects. Its value ranges from 1.0 to 1.5. The higher the frequency of dynamic object passage and the longer the dwell time, the larger the coefficient. For example, the dynamic image coefficient of model points in passage areas frequently traversed by pedestrians is 1.3.

[0163] Subpixel corner detection algorithms are algorithms that can detect subpixel-level (less than 1 pixel) corners in an image. This embodiment uses an improved Harris corner detection algorithm, with a detection accuracy of up to 0.1 pixels.

[0164] Image feature points are points in an image that have obvious characteristics, such as corner points, edge intersections, and texture key points. They are the core basis for registering images with 3D models.

[0165] Multi-view geometric calibration is a calibration method that calculates the camera's intrinsic parameter matrix (focal length, principal point coordinates, distortion coefficients) and extrinsic parameter matrix (rotation matrix, translation matrix) using image data from multiple viewpoints, and it adopts the Zhang Zhengyou calibration method.

[0166] The extrinsic parameter matrix describes the position and orientation relationship between the camera coordinate system and the world coordinate system (the spatial topology model coordinate system). It includes the rotation matrix R (3×3) and the translation matrix T (3×1), in the format of... .

[0167] The three-dimensional coordinate system is a right-handed rectangular coordinate system used in the spatial topology model. The origin (0,0,0) is located at the southwest corner of the ground of the venue. The x-axis is along the east-west direction, the y-axis is along the north-south direction, and the z-axis is perpendicular to the ground and pointing upwards.

[0168] Initial compensation vector The model point space compensation vector, initially calculated based on image data from a single visible light camera and Kalman filter residual data, forms the basis for subsequent comprehensive compensation vectors.

[0169] The compensation factor is a parameter used to characterize the compensation intensity in the initial compensation vector, with a value range of 0 - 0.1. It is determined by the matching accuracy between the image feature points and the model points. The higher the matching accuracy, the larger the compensation factor.

[0170] The factor set is a set composed of all compensation factors of the same model point under different visible light cameras. For example, if the compensation factors of model point P under 3 cameras are 0.08, 0.07, and 0.09 respectively, the factor set is {0.08, 0.07, 0.09}.

[0171] The undetermined coefficient is calculated based on the factor set and reflects the compensation requirement coefficient of the model point. It is calculated through the mean and variance of the factor set. The formula is r = (mean / variance) × 0.1, with a value range of 0 - 1. Set r0 = 0.8 as the determination threshold for the point to be optimized.

[0172] The point to be optimized is a model point with an undetermined coefficient r < r0 (0.8), that is, a model point with a relatively high compensation requirement and insufficient current compensation accuracy, which needs to be further adjusted.

[0173] The unit box is a two-dimensional square box (ignoring the z-axis) drawn with the model point as the center, used to select adjacent model points around the model point. Its unit box size is 0.5m × 0.5m, including the central model point and 8 adjacent model points around it.

[0174] The first adjustment factor is an adjustment factor calculated based on the undetermined coefficients of adjacent model points within the unit box. The formula is f1 = (the number of points with r < r0 within the unit box / the total number of points within the unit box) × 0.5, with a value range of 0 - 0.5, and is used to correct the initial compensation vector of the model point.

[0175] The minimum edge enclosure is the smallest polygon enclosure area drawn around all points to be optimized, which can include all points to be optimized and is used to analyze the distribution characteristics of the points to be optimized.

[0176] The second adjustment factor is an adjustment factor calculated based on the characteristic parameters of the minimum edge enclosure area, with a value range of 0 - 0.5, and is used to further correct the initial compensation vector and improve the compensation accuracy.

[0177] In this embodiment, for example, 3 visible light cameras synchronously shoot the venue images, and the improved Harris sub-pixel corner detection algorithm is used to extract feature points for each image. For example, 2000 feature points are extracted from an image of the exhibition area, and the sub-pixel coordinates (u, v) of each feature point are accurate to 0.1 pixel;

[0178] The Zhang-Zhengyou calibration method is used to perform multi-view geometry calibration on the cameras to obtain the internal parameter matrix K and the external parameter matrix , , where the focal length is 1000 pixels, the principal point coordinates are (1280, 960), the external parameter matrix R is a 3×3 rotation matrix, and T is a 3×1 translation matrix [5, 10, 15];

[0179] Through the camera imaging model (X = (u - u0) / f × Z, Y = (v - v0) / f × Z, where Z is the depth information of the feature point provided by the lidar data), the sub-pixel coordinates of the image feature points are converted into the three-dimensional coordinates (X, Y, Z) of the spatial topology model, realizing the mapping of the feature points to the three-dimensional coordinate system.

[0180] The first adjustment factor is determined for non-optimization points. A 0.5m×0.5m unit box is drawn with this model point as the center, and 9 model points (center + 8 adjacent points) within the unit box are traversed, and the number of points where r < r0 = 0.8 is counted. Suppose there are 2 points in the unit box where r < 0.8, then the first adjustment factor f1 = (2 / 9) × 0.5 ≈ 0.111.

[0181] The second adjustment factor is determined by drawing the smallest edge enclosing area for all optimization points, and calculating the point density index, point uniformity and functional correlation index of this area.

[0182] The beneficial effects of the above technical solution are: by constructing a dynamic weight model, accurately extracting image feature points and mapping them, and calculating the compensation vector adjustment factor in multiple dimensions, the refined calculation of the spatial compensation vector is realized, which can fully integrate the advantages of multi-source data, accurately correct the position deviation of the model points, effectively improve the pertinence and accuracy of spatial compensation, and provide reliable vector data support for the high-precision adjustment of the subsequent spatial topology model.

[0183] The present invention provides an application method of multi-sensor fusion technology in the three-dimensional scanning of a venue, constructing an initial compensation vector for each model point under the same visible light camera , including:

[0184] , where is the sub-pixel level three-dimensional coordinate vector of the image feature points under the same visible light camera, is the three-dimensional coordinate vector of the original points of the spatial topology model, is the residual vector of the Kalman filter dynamic compensation model, ]>is the image mapping weight, is the dynamic compensation weight adjusted based on the dynamic response coefficient.

[0185] In this embodiment, the image mapping weight w1 is a weight coefficient for adjusting the influence degree of the image feature point mapping deviation on the initial compensation vector, with a value range of 0 - 1, determined by the registration accuracy of the image and the three-dimensional model, and w1 = 0.6 is set.

[0186] In this embodiment, It is a three-dimensional coordinate vector obtained by transforming the sub-pixel two-dimensional coordinates of image feature points through sub-pixel corner detection and multi-view geometric calibration. The format is (X,Y,Z), such as (20.003,30.002,5.004).

[0187] In this embodiment, It is the three-dimensional coordinate vector of the original model point in the spatial topology model without compensation and correction, in the format (X,Y,Z), which is constructed from the initial three-dimensional contour data, such as (20,30,5).

[0188] The dynamic compensation weight w2 is used to adjust the weight coefficient of the influence of the Kalman filter residual on the initial compensation vector. The value range is 0-1 and is positively correlated with the interference intensity of the dynamic object. The stronger the interference, the larger w2 is. Set w2=0.4 and w1+w2=1.

[0189] In this embodiment, It is a vector of differences between the state estimates and actual observations of the Kalman filter model, in the format of This reflects the modeling error caused by dynamic disturbances, such as (0.001, 0.002, 0.001).

[0190] The beneficial effects of the above technical solution are: it clarifies the calculation logic and parameter values ​​of the initial compensation vector, and by reasonably allocating the image mapping weight and dynamic compensation weight, it fully integrates the position information of image feature points and the dynamic error correction information of Kalman filtering. The calculated initial compensation vector can accurately reflect the position deviation of model points, providing scientific and reliable basic data for the subsequent introduction of adjustment factors and the final determination of spatial compensation vector.

[0191] This invention provides a method for applying multi-sensor fusion technology in 3D scanning of venues, adjusting the corresponding spatial topology model based on the spatial compensation vector of each model point, including:

[0192] Fine-grained feature extraction is performed on the spatial compensation vectors of all model points, using the following formula:

[0193] ,in, This is the default feature representation for spatial compensation. For multi-scale interactive attention modules, i.e., through parameters Adapt to compensation vector features of different scales; For adaptive residual structure, i.e. through coefficients Dynamically adjust residual weights;

[0194] Based on the extracted features, all model points are divided into clusters according to accuracy requirements, and a scene-cluster dynamic priority mapping table is constructed in combination with the venue application scenario.

[0195] Each precision requirement cluster is treated as an independent adjustment unit. High precision requirement clusters are adjusted using point cloud-topology residual dual verification, medium precision requirement clusters are adjusted using point cloud residual single verification, and low precision requirement clusters are adjusted using simplified verification. At the same time, different precision requirement clusters in the corresponding application scenarios are adjusted according to the scenario-cluster dynamic priority mapping table.

[0196] Redundant nodes are removed from the adjusted topology model to obtain an effective topology model.

[0197] In this embodiment, fine-grained feature extraction is the process of extracting subtle features (such as vector magnitude, direction changes, and distribution patterns) of the spatial compensation vector, which is used to distinguish the differences in accuracy requirements of different model points.

[0198] The default spatial compensation feature representation It is the initial feature vector of the preset spatial compensation vector, containing parameters such as vector magnitude, direction angle, and variance, in the format of For example, (0.005, 30°, 45°, 60°, 0.00001).

[0199] In this embodiment, the multi-scale interactive attention module (MSIA) can adaptively adapt to attention mechanism modules that compensate vector features at different scales, through parameters. Adjusting the weights of features at different scales, and in this embodiment =0.7, emphasizing mesoscale features.

[0200] Adaptive Residual Structure (ARS) is a network structure that dynamically adjusts residual weights through coefficients. To control the impact of residuals on feature extraction results, this embodiment... =0.3, balancing the original features and residual features.

[0201] In this embodiment, It is a compensation vector feature obtained after processing by the MSIA module and ARS structure, containing richer and more subtle feature information, and is used for cluster partitioning with high accuracy requirements.

[0202] Accuracy requirement clusters are sets of model points with the same or similar accuracy requirements, divided according to fine-grained features. In this embodiment, they are divided into three categories: high-precision requirement clusters, medium-precision requirement clusters, and low-precision requirement clusters.

[0203] The scenario-cluster dynamic priority mapping table records the correspondence between the priority of clusters and the accuracy requirements of different application scenarios in the venue, as shown in Table 2 below:

[0204] Table 2 Scenario-Cluster Dynamic Priority Mapping Table

[0205]

[0206] In this embodiment, the independent adjustment unit treats each accuracy requirement cluster as a separate adjustment object and uses a differentiated adjustment strategy to optimize the model.

[0207] Point cloud-topology residual dual verification is an adjustment verification method adopted for high-precision clusters. It verifies the residuals of point cloud data and topology model at the same time to ensure adjustment accuracy. The residual threshold is set to 0.003m.

[0208] Point cloud residual single verification is an adjustment verification method used for clusters with medium accuracy requirements. It only verifies the residuals of point cloud data, and the residual threshold is set to 0.005m.

[0209] Simplified adjustment is an adjustment method used for low-precision clusters. It does not require complex verification. It directly adjusts the coordinates of model points based on the spatial compensation vector. The allowable adjustment error range is 0.01-0.02m.

[0210] Priority adjustment is based on a scene-cluster dynamic priority mapping table, which adjusts clusters with different precision requirements in order of priority. Clusters with higher priority are adjusted first to ensure the precision requirements of critical scenes.

[0211] Redundant nodes are duplicate or invalid nodes in a spatial topology model that have little or no impact on the model's accuracy, such as adjacent nodes less than 0.01m apart or isolated nodes outside the venue's boundaries.

[0212] Redundant node removal uses a distance threshold-based algorithm to remove redundant nodes from the model, reducing the amount of model data and improving model running efficiency.

[0213] Precision requirement cluster partitioning: based on fine-grained features The amplitude and variance parameters are used to divide the precision requirement clusters:

[0214] High-precision requirement cluster: amplitude ≥ 0.008m and variance ≤ 0.00001, corresponding to key structures such as load-bearing walls and equipment installation areas of the venue, such as model points with amplitude 0.01m and variance 0.000005;

[0215] Medium precision requirement cluster: amplitude 0.003-0.008m and variance 0.00001-0.00005, corresponding to the audience area, exhibition area and other conventional areas of the venue, such as model points with amplitude 0.005m and variance 0.00003;

[0216] Low-precision requirement cluster: amplitude ≤ 0.003m and variance ≥ 0.00005, corresponding to non-critical areas such as passageways and corners of the venue, such as model points with amplitude 0.002m and variance 0.00006.

[0217] In this embodiment, model tuning determines the current application scenario as equipment installation calibration. According to the mapping table, the high-precision requirement cluster has priority 1, medium precision 2, and low precision 3.

[0218] High-precision cluster adjustment: Point cloud-topology residual dual verification is adopted. After applying the spatial compensation vector to the model points, the point cloud residual and topology residual are calculated. If both are ≤0.003m, the adjustment is effective; if they exceed the threshold, the compensation vector is recalculated and adjusted.

[0219] For medium-precision cluster adjustment: point cloud residuals are used for verification. If the point cloud residuals are ≤0.005m after adjustment, it is considered effective.

[0220] Low-precision cluster adjustment: A simplified adjustment is adopted, directly correcting the model point coordinates based on the spatial compensation vector, with an allowable error of 0.01-0.02m.

[0221] The beneficial effects of the above technical solution are: it achieves accurate classification of model points through fine-grained feature extraction, formulates dynamic priority adjustment strategies in combination with scenario requirements, adopts differentiated adjustment and verification methods, ensures high accuracy in key areas while taking into account the efficiency of model adjustment, reduces the amount of model data by eliminating redundant nodes, and finally obtains an effective topology model with high accuracy, lightweight and strong adaptability, which can meet the application needs of different scenarios.

[0222] This invention provides a method for applying multi-sensor fusion technology in 3D scanning of venues, including drawing minimum edge bounding diagrams for all points to be optimized and determining a second adjustment factor, comprising:

[0223] The ratio of the number of points to be optimized within the enclosed area to the area of ​​the region is determined to obtain the point density index;

[0224] Quantify the point uniformity of the point to be optimized within the enclosing region;

[0225] Analyze the spatial overlap between the surrounding area and different functional areas of the venue to obtain functional correlation indicators;

[0226] Based on the point density index, point uniformity, and functional correlation index, a second adjustment factor is obtained.

[0227] In this embodiment, the enclosing region is the smallest edge enclosing region drawn around all the points to be optimized. In this embodiment, it is a polygonal region with coordinates ranging from (x1, y1, x2, y2), such as (15, 25, 35, 45).

[0228] The point density index is the ratio of the number of points to be optimized within an enclosed region to the area of ​​the enclosed region, measured in units of... This reflects the density of points to be optimized, such as the area of ​​the enclosed region being 100. There are 50 points to be optimized, and the point density index is 50 / 100 = 0.5. .

[0229] Point uniformity refers to the evenness of the distribution of points to be optimized within the enclosed area. It is quantified by calculating the variance of the coordinates of the points to be optimized. The smaller the variance, the better the uniformity. In this embodiment, the variance ranges from 0 to 10. For example, a variance of 2.5 indicates good uniformity.

[0230] The functional correlation index is the spatial overlap between the enclosed area and different functional zones of the venue, specifically the ratio of the area of ​​the overlapping portion of the enclosed area and a certain functional zone to the total area of ​​the enclosed area. The value ranges from 0 to 1. For example, if the overlapping area between the enclosed area and the exhibition area is 80... Total area 100 Functional correlation index = 80 / 100 = 0.8.

[0231] The second adjustment factor is an adjustment factor obtained by combining the point density index, point uniformity, and functional correlation index. Its value range is 0-0.5, and the formula is f2=(point density index × 0.4 + normalized point uniformity × 0.3 + functional correlation index × 0.3). The normalized point uniformity is calculated by (preset maximum variance - actual variance) / preset maximum variance. In this embodiment, the preset maximum variance is 10 to ensure that the value range of the second adjustment factor is between 0 and 0.5, which can reasonably balance the correction effect of each dimension on the compensation vector.

[0232] The beneficial effects of the above technical solution are: by using three dimensions—point density index, point uniformity, and functional correlation index—the distribution characteristics of the points to be optimized and the correlation with the functions of the venue can be fully quantified. The calculated second adjustment factor can accurately match the actual situation of the area to be optimized, avoiding the limitations of single-dimensional correction. This helps to improve the accuracy and rationality of spatial topology model adjustment, especially for the correction effect of points to be optimized in key functional areas.

[0233] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for applying multi-sensor fusion technology in 3D scanning of venues, characterized in that, include: Step 1: Pre-scan the target venue based on LiDAR, and obtain initial three-dimensional contour data and construct a spatial topology model based on the output quality determined by the output signal of LiDAR in each venue unit and the unit attributes of the corresponding venue unit determined by the venue construction drawing; Step 2: Based on real-time monitoring of the motion trajectory of dynamic objects inside the target venue and the change in lighting intensity inside the venue using a combination of multiple sensors, construct a Kalman filter dynamic compensation model; Step 3: Obtain the shooting results of each venue unit by cameras with different visible light, and perform point mapping compensation on the spatial topology model based on the shooting results of the same visible light camera. Combine the dynamic compensation model of Kalman filter with the dynamic compensation matching relationship of each model point to construct the spatial compensation vector of each model point. Step 4: Adjust the corresponding spatial topology model based on the spatial compensation vector of each model point to obtain an effective topology model and output it.

2. The method for applying multi-sensor fusion technology in 3D scanning of venues according to claim 1, characterized in that, Also includes: When pre-scanning the target venue using lidar, the environmental perception module collects the light intensity, dust concentration and air turbulence coefficient in the venue in real time, and identifies suspected occlusion areas from the point cloud data output by lidar in real time to obtain the current occlusion distribution. The current occlusion distribution is divided into occlusion degree categories, and the classification results are firstly labeled according to the scanning order. At the same time, the total area of ​​the same occlusion degree is determined, and a second label is performed according to the concentration location of each total area and the scanning angle of the center point of the concentration location during the scanning process, and combined with the minimum and maximum angles based on the scanning order for each location point in the concentration location. Based on the first and second order calibrations, and combined with the distribution uniformity, the re-scanning process is determined, and the transmission power during re-scanning is n0 times that of conventional scanning, so as to realize environmental adaptation and occlusion compensation acquisition of the pre-scanned point cloud.

3. The method for applying multi-sensor fusion technology in 3D scanning of venues according to claim 1, characterized in that, Acquire initial 3D contour data and construct a spatial topology model, including: The pre-scanning result is regarded as the first three-dimensional contour data, and the first three-dimensional contour data is divided into several sub-data according to the smallest contour unit. The basic contour is drawn for each sub-data to obtain the sub-contour. A two-dimensional matrix is ​​constructed based on the output quality determined by the output signal of the lidar in each venue unit and the unit attributes of the corresponding venue unit determined by the venue construction plan; Based on the location comparison relationship, determine all the sub-contours involved in each venue unit, and construct the offset matrix of the corresponding venue unit according to the offset coefficient between the corresponding venue unit and each corresponding sub-contour; The offset matrix is ​​time-aligned with the set of scanning angle changes of the lidar in the corresponding venue unit, and coupled with each sub-contour involved in the corresponding venue unit to obtain the initial three-dimensional contour data. Determine the structure to which each sub-contour belongs and its corresponding two-dimensional matrix in the venue construction drawing, and determine the construction accuracy of the corresponding sub-contour; According to the construction accuracy, the unit spatial modeling shape based on each sub-contour is obtained from the accuracy-design lookup table, and a spatial topology model is constructed based on the initial three-dimensional contour data.

4. The method for applying multi-sensor fusion technology in 3D scanning of venues according to claim 1, characterized in that, The multi-sensor combination includes an infrared thermal imaging sensor, a millimeter-wave sensor, a vision sensor, and a light sensor. The infrared thermal imaging sensor and the millimeter-wave sensor generate the first motion trajectory of the first object and the second motion trajectory of the second object, respectively. The vision sensor collects the contour features of the dynamic object to distinguish the object type. The light sensor is arranged in a grid on the top of the venue to collect the absolute value of the light intensity and the rate of change per second within the grid unit in real time. Based on the similarity between the first and second motion trajectories, the infrared thermal imaging sensor and the millimeter-wave sensor are correlated and matched to monitor targets. Simultaneously, for different types of dynamic objects, differentiated motion feature baselines are preset and based on… Obtain the motion trajectory coordination coefficients of each dynamic object. ,in, This is a preset synergy coefficient correction factor; The preset deviation threshold; This represents the deviation between the real-time motion parameters and the characteristic baseline. In monitoring changes in indoor lighting intensity, the attenuation rate of lighting intensity within the object's projection area is calculated by combining the real-time position coordinates of dynamic objects, and based on... Correction factor for generating changes in light intensity ,in, The preset illumination correction factor, The light intensity attenuation rate; The preset attenuation rate threshold; Based on the motion trajectory coordination coefficient With correction factor Adjust the noise covariance of the model filtering process; The dynamic object contour features acquired by the visual sensor are used as observation constraints in the filtering model, and a Kalman filter dynamic compensation model is constructed by combining the adjusted covariance.

5. The method for applying the multi-sensor fusion technology in 3D scanning of venues according to claim 1, characterized in that, Step 3 includes: A dynamic weight model of image-topology unit is constructed, and the weight is positively correlated with the unit structure complexity of the spatial topology model and the spectral responsiveness of building materials. In addition, dynamic image coefficients are assigned to the model points in the region traversed by the dynamic object in the Kalman filter dynamic compensation model. When performing point mapping compensation on the spatial topology model based on the shooting results of the same visible light camera, a sub-pixel corner detection algorithm is used to extract image feature points, and the external parameter matrix of the camera and the lidar is obtained based on multi-view geometric calibration, and the image feature points are mapped to the three-dimensional coordinate system of the spatial topology model. Construct the initial compensation vector for each model point under the same visible light camera. ; Iterate through the initial compensation vectors of each model point under different visible light cameras. The compensation factors are determined, and a factor set is constructed to analyze the undetermined coefficients of the corresponding model points; When the undetermined coefficient is less than r0, the corresponding model point is determined as the point to be optimized; Otherwise, draw the unit frame to the corresponding model point, and determine the undetermined coefficients of the remaining points in the unit frame other than the corresponding model point, and determine the first adjustment factor based on the points in the unit frame that are less than r0. Draw a minimum edge bounding plot for all points to be optimized, and determine the second adjustment factor; Based on the first adjustment factor and the second adjustment factor, for all model points After comprehensive processing, the spatial compensation vectors of the corresponding model points are obtained.

6. The method for applying the multi-sensor fusion technology in 3D scanning of venues according to claim 5, characterized in that, Construct the initial compensation vector for each model point under the same visible light camera. ,include: ,in, The image feature points under the same visible light from the camera are represented by sub-pixel-level three-dimensional coordinate vectors. The three-dimensional coordinate vector of the original point in the spatial topology model. Let be the residual vector of the Kalman filter dynamic compensation model. Image mapping weights, This refers to the dynamic compensation weight adjusted based on the dynamic response coefficient.

7. The method for applying multi-sensor fusion technology in 3D scanning of venues according to claim 1, characterized in that, The corresponding spatial topology model is adjusted based on the spatial compensation vector of each model point, including: Fine-grained feature extraction is performed on the spatial compensation vectors of all model points, using the following formula: ,in, This is the default feature representation for spatial compensation. For multi-scale interactive attention modules, i.e., through parameters Adapt to compensation vector features of different scales; For adaptive residual structure, i.e. through coefficients Dynamically adjust residual weights; Based on the extracted features, all model points are divided into clusters according to accuracy requirements, and a scene-cluster dynamic priority mapping table is constructed in combination with the venue application scenario. Each precision requirement cluster is treated as an independent adjustment unit. High precision requirement clusters are adjusted using point cloud-topology residual dual verification, medium precision requirement clusters are adjusted using point cloud residual single verification, and low precision requirement clusters are adjusted using simplified verification. At the same time, different precision requirement clusters in the corresponding application scenarios are adjusted according to the scenario-cluster dynamic priority mapping table. Redundant nodes are removed from the adjusted topology model to obtain an effective topology model.

8. The method for applying the multi-sensor fusion technology in 3D scanning of venues according to claim 5, characterized in that, Minimum edge bounding is plotted for all points to be optimized, and the second adjustment factor is determined, including: The ratio of the number of points to be optimized within the enclosed area to the area of ​​the region is determined to obtain the point density index; Quantify the point uniformity of the point to be optimized within the enclosing region; Analyze the spatial overlap between the surrounding area and different functional areas of the venue to obtain functional correlation indicators; Based on the point density index, point uniformity, and functional correlation index, a second adjustment factor is obtained.

Images