An indoor personnel positioning method and system based on multi-modal perception
By combining multimodal sensing technology with ultra-wideband, millimeter-wave radar and thermal imaging data, indoor ray paths are reconstructed and preprocessed, solving the problems of positioning accuracy and stability in complex building environments and achieving high-precision positioning under fire smoke and water mist interference.
Patent Information
- Application Number
- CN202511759553.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-27
- Publication Date
- 2026-02-06
- Estimated Expiration
- 2045-11-27
AI Technical Summary
Existing indoor personnel positioning methods suffer from low positioning accuracy and insufficient stability in complex building environments, especially under conditions of fire smoke, water mist interference, and multipath effects, making it difficult to maintain positioning accuracy.
Using multimodal sensing technology, combined with ultra-wideband channel impulse response, millimeter-wave radar data and thermal imaging data, indoor ray paths are reconstructed through bimodal geometric mapping to generate an occupancy grid map. Water mist distribution intensity is collected for preprocessing, and motion prior information and boundary constraint information are extracted. Finally, the fusion solution is performed on the occupancy grid map to output the location information of the object to be located.
It maintains high positioning accuracy and stability in environments with dense smoke, water mist interference, and significant multipath effects, making it suitable for high-risk scenarios such as fire rescue.
Smart Images

Figure CN121252813B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of indoor positioning, and in particular to an indoor personnel positioning method and system based on multi-modal perception. BACKGROUND
[0002] Some indoor personnel positioning methods rely on a single sensing mode, which often has insufficient applicability in complex building environments. For example, positioning methods based on wireless signals have low positioning accuracy in space occlusion and multipath environments, and are prone to produce false position points; positioning methods based on vision or infrared imaging have degraded imaging quality in conditions such as smoke and water mist, and the target outline is blurred, making it difficult to extract effective features; positioning methods based on millimeter wave radar have insufficient signal stability in strong scattering interference environments, and the output results drift.
[0003] To solve the above problems, the present application designs an indoor personnel positioning method and system based on multi-modal perception. SUMMARY
[0004] The technical problem to be solved by the present application is to provide an indoor personnel positioning method and system based on multi-modal perception to solve the problems of the prior art. The indoor ray path is reconstructed and an occupancy grid map is generated by obtaining the ultra-wideband channel impulse response, millimeter wave radar data and thermal imaging data of each layer, combining building information for double-base geometric mapping. Further, the water mist distribution intensity generated by the spray device is collected, the millimeter wave data is phase corrected and beam domain purified, and the thermal imaging data is flow filtered and flicker suppressed to obtain preprocessed data. On this basis, motion prior information and boundary constraint information are extracted, and fusion is achieved by quality conservation update and normalization synthesis in the occupancy grid map, and finally the position information and motion direction of the object to be positioned are output. The present application scheme can maintain positioning stability and accuracy under the interference of fire smoke, water mist and multipath effect, and is suitable for high-risk scenarios such as fire rescue.
[0005] To achieve the above purpose, the present application provides the following technical scheme:
[0006] An indoor personnel positioning method based on multi-modal perception is applied to indoor positioning of an object to be positioned in a multi-story building, each floor of the multi-story building comprising a sensor assembly arranged around, the sensor assembly comprising at least an ultra-wideband node, a millimeter wave radar and a thermal imaging camera in communication with each other, and the method comprising:
[0007] Obtaining channel impulse responses, millimeter wave data and thermal imaging data of each layer;
[0008] According to the channel impulse response, combining the preset building information, reconstructing the indoor elliptical ray path by double-base geometric mapping to obtain an occupancy grid map;
[0009] Collect water mist information, preprocess the millimeter wave data and thermal imaging data according to the water mist information, calculate motion prior information according to the preprocessed millimeter wave data, and calculate boundary constraint information according to the preprocessed thermal imaging data;
[0010] Fuse the motion prior information and the boundary constraint information in the occupancy grid map to obtain position information of the object to be positioned.
[0011] The indoor elliptical ray path is reconstructed by double-base geometric mapping to obtain an occupancy grid map, comprising:
[0012] The transmitting end and the receiving end of the ultra-wideband nodes in mutual communication are taken as a pair of double-base points, the isochronous delay elliptical trajectory corresponding to the pair of double-base points is calculated according to the propagation time of the corresponding time delay component in the channel impulse response, and the isochronous delay elliptical trajectory is summarized to obtain an elliptical ray map;
[0013] The elliptical ray map is mapped to a preset three-dimensional building information coordinate system, and the energy weight of the trajectory segment passing through the voxel is allocated to obtain the occupancy signal strength of each voxel. The initial occupancy grid map is generated based on each voxel, wherein the grid in which the occupancy signal strength corresponding to the voxel is greater than a preset voxel threshold is an occupancy grid, and the grid is set according to the building information;
[0014] According to the entity structure in the building information, the occupancy grid in the initial occupancy grid map is corrected to obtain an occupancy grid map.
[0015] The water mist information is water mist distribution intensity, and the collection method of the water mist distribution intensity is as follows:
[0016] Obtain the spraying parameters of the spraying equipment on each floor in the building information, wherein the spraying parameters include the opening state of the spray head, the spraying flow and the spraying pressure;
[0017] According to the spraying parameters, the arrangement position of the spraying equipment and the preset spraying characteristic curve, the water mist distribution intensity of each floor is calculated.
[0018] The millimeter wave data is preprocessed according to the water mist information, comprising:
[0019] According to the building information, a static reference reflector in the corresponding floor is selected as a calibration target;
[0020] The common phase error and the group delay deviation corresponding to the calibration target are calculated in combination with the distance-angle-velocity data of each frame in the millimeter wave data;
[0021] The phase structure function is calculated according to the water mist information, the common phase error and the group delay deviation;
[0022] interpolating the millimeter wave data according to the phase structure function to obtain correction data, wherein the interpolation is used to recover the coherence of the millimeter wave data;
[0023] transforming the correction data to a beam domain by a subspace decomposition algorithm, purifying the correction data in the beam domain by a preset decomposition model to obtain purified millimeter wave beam domain data, wherein the decomposition model includes a low-rank background term and a sparse target term, the low-rank background term corresponds to water vapor volume scattering and non-directional echoes, and the sparse target term corresponds to human body reflection echoes;
[0024] back-projecting the purified millimeter wave beam domain data to obtain preprocessed millimeter wave data.
[0025] preprocessing the thermal imaging data according to the water vapor information, including:
[0026] calculating a spatial gradient of the water vapor distribution intensity according to the arrangement position corresponding to the water vapor distribution intensity, and calculating a local flow field of the spatial gradient in combination with the position of the thermal imaging camera;
[0027] calculating a streamline direction of each pixel in the thermal imaging data according to a principal component streamline of the local flow field;
[0028] extracting a time sequence of each pixel in the streamline direction, and performing time domain filtering processing on the time sequence, wherein a filtering window length is calculated according to the water vapor distribution intensity of a corresponding position;
[0029] calculating a flicker frequency band by frequency spectrum analysis according to the time sequence after the time domain filtering processing, performing suppression processing on the flicker frequency band, and summarizing the time sequence after the suppression processing of each pixel to obtain preprocessed thermal imaging data, wherein the suppression processing reduces the energy amplitude of the flicker frequency band by a notch filter, and a parameter of the notch filter is calculated according to the water vapor distribution intensity of a corresponding position.
[0030] calculating motion prior information according to the preprocessed millimeter wave data, including:
[0031] performing range-angle-velocity three-dimensional imaging on the preprocessed millimeter wave data to obtain a three-dimensional data cube;
[0032] setting a fall velocity suppression band associated with the water vapor information in a velocity dimension, and detecting the three-dimensional data cube according to the fall velocity suppression band to obtain a motion candidate unit;
[0033] performing connected domain clustering on the motion candidate unit in a range-angle two-dimensional plane to obtain a target scattering cluster;
[0034] For each target scattering cluster, the radial velocity, azimuth angle and distance parameters are calculated, and combined with adjacent frames, the velocity vector of the target scattering cluster and the confidence of the velocity vector corresponding to the velocity vector are calculated by multi-target trajectory estimation, and the velocity vector and the corresponding confidence are taken as the motion prior information.
[0035] The boundary constraint information is calculated according to the preprocessed thermal imaging data, including:
[0036] Multi-scale gradient detection and region growing are performed on the preprocessed thermal imaging data to extract temperature mutation edges, and a candidate boundary contour is obtained;
[0037] The candidate boundary contour is screened according to the preset temperature threshold and contrast threshold, and a contour region meeting the human radiation characteristics is obtained;
[0038] The contour region is subjected to morphological processing and connected component analysis to generate a closed boundary curve, and the boundary constraint information is generated according to the closed boundary curve.
[0039] The motion prior information and the boundary constraint information are fused and solved in the occupancy grid map, including:
[0040] The projection of the boundary constraint information in the occupancy grid map is taken as a boundary mask, and the occupancy density map is obtained by intersecting the occupancy grid;
[0041] The occupancy density map is subjected to voxel-level mass conservation advection update according to the motion prior information to obtain the evidence density corresponding to the occupancy grid, wherein the mass conservation advection update calculates the update value of the occupancy grid through a backtracking sampling rule along the velocity direction, and the backtracking sampling rule includes inhibiting the flow of walls and disconnected regions and enhancing the flow at doorways and channel connections;
[0042] The evidence density and the occupancy grid map are normalized and synthesized to obtain an updated occupancy grid map, wherein the synthesis weight of the normalization and synthesis is calculated according to the velocity confidence and the boundary consistency;
[0043] A plurality of peak regions are selected in the updated occupancy grid map, the centroid of each peak region is calculated as the position information of the object to be positioned, and the principal axis direction of each peak region is output as the motion direction.
[0044] An indoor personnel positioning system based on multi-modal perception, the system comprising:
[0045] A perception acquisition module is configured to acquire channel impulse responses of ultra-wideband links, millimeter wave data of millimeter wave radars and thermal imaging data of thermal imaging cameras on each floor of a multi-story building, and receive building information and spraying parameters of a spraying device to calculate water mist distribution intensity;
[0046] a grid reconstruction module, configured to perform double-base geometric mapping according to the channel impulse response and the building information, reconstruct an indoor elliptical ray path and generate an occupancy grid map, and correct millimeter wave data according to the water mist distribution intensity and beam domain purification, and perform time domain filtering and flicker band suppression on thermal imaging data along the flow direction, to output preprocessed millimeter wave data and thermal imaging data;
[0047] a constraint extraction module, configured to calculate motion prior information from the preprocessed millimeter wave data, and extract a closed boundary curve from the preprocessed thermal imaging data to generate boundary constraint information;
[0048] a fusion positioning module, configured to fuse and solve the motion prior information and the boundary constraint information in the occupancy grid map, including voxel-level mass conservation advection update and normalized synthesis, and determine the centroid and principal axis of a peak region in the updated occupancy grid map, to output position information and motion orientation of an object to be positioned.
[0049] The constraint extraction module comprises:
[0050] a motion prior unit, configured to perform three-dimensional imaging and connected domain clustering on the preprocessed millimeter wave data, calculate radial velocity, azimuth angle and distance parameters of a target scattering cluster, and output a velocity vector in combination with adjacent frame trajectory estimation, to generate motion prior information;
[0051] a boundary generation unit, configured to perform multi-scale edge detection and region growing on the preprocessed thermal imaging data, filter out a contour region conforming to human body radiation characteristics, and generate a closed boundary curve through morphological processing and connected domain analysis, to obtain boundary constraint information.
[0052] Compared with the prior art, the application has the beneficial effects that:
[0053] The application can maintain high positioning accuracy and stability in an environment with smoke diffusion, water mist interference and significant multipath effect, by fusing ultra-wideband, millimeter wave radar and thermal imaging data, and performing preprocessing and constraint correction in combination with building information and water mist distribution intensity. BRIEF DESCRIPTION OF DRAWINGS
[0054] Other features, objects and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments, made with reference to the accompanying drawings:
[0055] Figure 1 An exemplary application scenario provided for the embodiments of the application is shown in the following figure:
[0056] Figure 2 A flowchart of an indoor personnel positioning method based on multi-modal perception provided for the embodiments of the application is shown in the following figure:
[0057] Figure 3 An isochronous delay ellipse trajectory diagram provided for an embodiment of the present application;
[0058] Figure 4 An initial occupancy grid diagram provided for an embodiment of the present application;
[0059] Figure 5 An occupancy grid diagram provided for an embodiment of the present application. DETAILED DESCRIPTION
[0060] The technical solutions in the embodiments of the present application will be apparently and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application.
[0061] In this document, reference to“an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase“in an embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. One of ordinary skill in the art can understand that an embodiment described herein can be incorporated in some embodiments and might be elicitly and implicitly understood that an embodiment described herein can be incorporated in other embodiments.
[0062] In the present embodiment, the method is directed to public buildings with complex spatial connectivity and complete fire facilities, including but not limited to high-rise commercial complexes, transportation hubs, hospital inpatient buildings, or large underground garages.
[0063] It can be understood that the present application can be applied to positioning and search and rescue work in actual fire disposal process.
[0064] It is easy to understand that the aforementioned buildings are not lack of cameras or network bandwidth, but are dominated by the unique physical environment of the fire scene:
[0065] After the spray is started, the water mist and steam quickly fill the corridor and shaft, and the visible light picture is first degraded and then covered by heat radiation and flame flicker;
[0066] The high-temperature updraft and positive pressure air supply change the ray propagation conditions, and the millimeter wave echo appears volume scattering lifting bottom and phase drift;
[0067] The smoke and door valve state differentiate the cross-floor signal fading, and the measurement results of a single sensor will be distorted within a few seconds.
[0068] In some optional embodiments, in response to such an environment, the conventional way of giving personnel a tag is not realistic, and the trapped person is usually empty-handed and easy to lose contact; target tracking by camera or single-mode radar is often disturbed by occlusion and artifacts. The difficulty encountered repeatedly in many drills and real events is that "what can be seen may not be able to pass, and what can pass may not be a real person", and cross-layer misjudgment and wall-penetrating false targets can mislead the search and rescue route and evacuation timing.
[0069] The processing logic of the present application does not pursue the stacking of computing power or the introduction of large-scale remote transmission, but returns to the certainty priori on the spot:
[0070] The model information of the building provides the accessibility boundary of the wall, door opening and staircase;
[0071] The spatial distribution intensity of the water mist is determined by the opening and closing of the spray head, flow and pressure in the floor partition, which is not only the cause of the degradation of vision and radar, but also a signal that can be quantified and used for correction;
[0072] Although the time delay response of the ultra-wideband link is affected by smoke and ventilation, the elliptical ray path corresponding to the double-base geometry still provides occupancy evidence consistent with the wall topology.
[0073] Under the prerequisite of the foregoing certainty priori, without changing the existing topology of the building, the present embodiment arranges the ultra-wideband nodes, millimeter wave radar and thermal imaging cameras on each floor as fixed perception pedestals:
[0074] The ultra-wideband time delay component reconstructs the elliptical ray in the building coordinate system and generates an initial occupancy grid; the water mist distribution intensity is derived from the spray parameters, which first cleans the frequency band flicker, phase drift and halo tail caused by water mist in millimeter wave and thermal imaging, and then extracts the velocity vector and closed boundary; finally, on the occupancy grid, the accessibility is taken as a hard constraint, and "motion can reach, boundary can envelope, and evidence can be conserved" as the fusion criterion, the position and floor determination is given.
[0075] It should be emphasized that the present embodiment focuses on local fusion of feature-level elements, and the returned information is abstract information such as position, boundary and confidence, rather than complete video stream or massive point cloud, so it does not depend on high-bandwidth transmission conditions; the system can still maintain positioning continuity in weak network or even temporary link failure scenarios, which meets the engineering reality of fire communication.
[0076] Reference Figure 1 , Figure 1 An exemplary application scenario is provided for the embodiment of the present application.
[0077] As Figure 1As shown, the example building is a multi-story structure, each floor is provided with rooms, and sensor components are arranged in each floor to obtain perception data of the corresponding floor. There are objects to be positioned in some rooms, which can be in different floors and rooms. Through the joint work of the aforementioned sensor components, multi-modal perception and positioning of the objects to be positioned inside the building can be achieved, which provides support for fire rescue or personnel search in emergency situations.
[0078] It can be understood that the sensor components in the present application include an ultra-wideband node, a millimeter wave radar, and a thermal imaging camera.
[0079] In some optional specific embodiments, the ultra-wideband node can be a separate node device fixedly installed on a wall or ceiling of a floor, or can be a communication unit integrated in a fire monitoring terminal or an Internet of Things base station.
[0080] In a specific arrangement, a ring arrangement can be arranged for each floor to achieve full coverage monitoring, or a partitioned key arrangement can be used to adapt to the detection needs of local space, and the present application does not make more limitations.
[0081] Next, a multi-modal perception-based indoor personnel positioning method provided by the embodiments of the present application is further expanded in combination with the drawings.
[0082] S1: Obtain channel impulse responses, millimeter wave data, and thermal imaging data of each floor;
[0083] In the present embodiment, first, the channel impulse responses, millimeter wave data, and thermal imaging data of each floor are obtained. The building is a multi-story structure as shown in Figure 1 The rooms of each floor are separated by walls, and sensor components including an ultra-wideband node, a millimeter wave radar, and a thermal imaging camera are arranged in the floor. The channel impulse responses are obtained by the ultra-wideband node detecting the communication link, the millimeter wave data comes from the distance, angle, and speed dimension measurements of the radar, and the thermal imaging data provides the temperature distribution of different spatial regions.
[0084] S2: According to the channel impulse responses, combined with the preset building information, reconstruct the indoor elliptical ray path through double-base geometry mapping to obtain the occupancy grid;
[0085] In the present embodiment, the transmitting end and the receiving end node are taken as double-base points, the isochronous delay elliptical trajectory is derived from the time delay component of the channel impulse responses, and is aligned with the three-dimensional building information to generate a ray path set covering the whole floor. The energy weight is assigned to the ray path passing through each voxel, so as to obtain the occupancy signal strength of each voxel, and whether it is an occupancy grid is determined according to the threshold.
[0086] Further, the wall and floor constraints in the building information are used to eliminate the occupancy cells that do not conform to the actual structure, thereby reducing the false targets introduced by reflection and multipath.
[0087] It is easy to understand that the processing logic of the embodiment can establish spatial occupancy information consistent with the structure under complex topology and limited visual angle conditions, and ensure a reliable environment base for subsequent fusion links.
[0088] S3: Collect water mist information, and pre-process the millimeter wave data and the thermal imaging data according to the water mist information;
[0089] In the embodiment, the water mist information is calculated from the nozzle parameters of the spraying equipment, including the opening state, the spraying flow and the spraying pressure, and the water mist distribution intensity in the space is estimated in combination with the spraying characteristic curve. Based on the water mist distribution intensity, the phase drift and volume scattering correction of the millimeter wave data are performed, and low-rank background and sparse target decomposition is performed in the beam domain to filter out the non-directional echoes generated by the water mist; the thermal imaging data is filtered in time sequence according to the flow field to suppress the flicker noise caused by the water mist particles, and the frequency band disturbance is suppressed by wave trapping.
[0090] S4: Calculate motion prior information according to the pre-processed millimeter wave data, and calculate boundary constraint information according to the pre-processed thermal imaging data;
[0091] In the embodiment, after three-dimensional imaging of the millimeter wave data, target scattering clusters are obtained through candidate cell detection and connected domain clustering, and velocity vectors and confidence are calculated in combination with adjacent frame trajectories to form motion prior information; the thermal imaging data is subjected to multi-scale edge extraction and region growing, and the contour region meeting the human body radiation characteristics is screened out, and then subjected to morphological analysis to generate a closed boundary curve as boundary constraint information.
[0092] S5: Fuse and solve the motion prior information and the boundary constraint information in the occupancy grid map to obtain position information of the object to be positioned;
[0093] In the embodiment, the fusion method is based on the mass conservation advection update and boundary consistency correction at the voxel level, the velocity vector is used to guide the transfer of occupancy evidence on the time axis, and the closed boundary constraint is used to prevent the evidence from spreading in the unreachable area. Finally, in the updated occupancy grid, the centroid of the high-confidence peak value region is selected as the target position, and the motion direction is output through the principal axis direction.
[0094] It is easy to understand that the fusion logic of the embodiment can maintain positioning continuity and reliability in the case of high smoke concentration, narrow channel or serious cross-floor fading, and solve the dilemma of wall-penetrating false targets and trajectory interruption in the strong interference fire field of traditional methods.
[0095] Before the specific technical content corresponding to the unfolding step is described, the embodiments of the present application need to be emphasized again.
[0096] The application object of the present application takes a multi-story public building with fire-fighting facilities as an example, and a sensor assembly is arranged on each floor along the corridor for continuous sensing, aiming to output the position and floor information without relying on any equipment carried by the searchee. Unlike the common direct discrimination of the target from a single sensor, the embodiments of the present application first take the deterministic condition of the building as a long-term effective priori, and then regard the actual situation on the scene as a quantifiable external variable, so as to convert the complex scene constraint into a calculable boundary and weight, which is particularly crucial for maintaining the explainability of the positioning conclusion.
[0097] Further, based on the foregoing priori, the embodiments of the present application take the time delay component of the ultra-wideband link as a clue, reconstruct a double-base isochronous delay elliptical trajectory in the building information model coordinates, form a ray coverage consistent with the structure, and then obtain initial occupancy evidence according to the path and energy distribution of the traversed voxel, without requiring extremely high density of node layout, the key is to determine the hard premise of accessibility or inaccessibility, so that the false high value introduced by multipath or blind angle is suppressed at the structure level; in order to avoid unnecessary constraints on the division of a specific floor, the type or installation position of the node, the voxel size, the weight threshold and the ray synthesis strategy can be set according to the site conditions, and the present application does not make more limitations.
[0098] Still further, considering the scene degradation caused by spraying and air supply and exhaust, the embodiments of the present application combine the spray head opening state, flow and pressure with the spray characteristic curve to deduce the water mist distribution intensity and general flow direction of each region, and accordingly, the working condition related preprocessing is first performed on the millimeter wave and thermal imaging.
[0099] It should be noted that this is not to increase the sensing complexity, but to directly quantify the cause of degradation and use it for correction, so that the speed and boundary of subsequent extraction are closer to the real target rather than the working condition artifacts. The water mist information can be calculated discretely according to the floor or partition, and the time update granularity and estimation method can be adjusted according to the device conditions, and the present application does not make more limitations.
[0100] Next, the technical content of the embodiments of the present application method about the occupancy grid is unfolded.
[0101] It can be understood that the method of the present application can be used for positioning on a floor in a multi-story building, or for positioning on each floor in a multi-story building, depending on the determination degree of the floor state, when the posterior probability of the target floor is improved to above the trigger threshold by the information such as air pressure difference, cross-floor link energy difference and millimeter wave elevation angle, only the planar reconstruction is performed on the floor; when the floor is uncertain or there is confusion between the same floor / upper and lower floors, the voxelized three-dimensional reconstruction is directly performed within the floor height range, so as to avoid the projection deviation caused by early misjudgment.
[0102] Those skilled in the art can understand that the triggering threshold can be set according to the on-site computing capability and the latency requirement, and the present application does not make more limitations.
[0103] For personnel positioning on a certain floor, the time delay component of the ultra-wideband link is projected into an equal-time delay ellipse and discretized to a grid unit in the BIM plane coordinate system of the floor, the energy weight is allocated according to the traversal length and the incident angle of the ray path in the unit, and the initial occupancy intensity map of the layer is obtained by superposition; then the connectivity correction is performed by using the structure mask of the wall and the door hole, and thresholding is performed, to form a two-dimensional occupancy grid map. The operation amount increases approximately linearly with the link, the time delay segment and the effective grid number, the memory occupation is low, and it is suitable for real-time updating at the edge; in the search stage of the known floor, a higher frame rate and more stable positioning continuity can be obtained.
[0104] For personnel positioning on each floor, the time delay component is corresponded to a double-base equal-time delay ellipsoidal surface in the three-dimensional BIM coordinate system, the traversal contribution of the cross-floor link to the upper and lower layers is taken into account, the energy is accumulated according to the traversal volume of the voxel and the geometric consistency, and a three-dimensional occupancy grid is obtained; then the voxel-level mask and connectivity constraint are performed with the floor and the vertical member as the hard boundary, and the occupancy slice of each layer is obtained by slicing according to the layer height, which is used for floor determination and positioning output. The calculation and storage of three-dimensional reconstruction increase with the number of voxels, the number of links and the time delay segment, and the complexity is significantly higher than that of two-dimensional, but it can provide more robust evidence when the floor is uncertain and the vertical channel interference is obvious.
[0105] In one example, the specific steps of S2 are as follows:
[0106] S2.1: Taking the transmitting end and the receiving end of the ultra-wideband nodes in mutual communication as a pair of double-base points, calculating the equal-time delay ellipse trajectory corresponding to the double-base point pair according to the propagation time of the corresponding time delay component in the channel impulse response, and obtaining an elliptical ray map by summarizing the equal-time delay ellipse trajectory;
[0107] Specifically, the pair of working ultra-wideband nodes are taken as double-base points in the three-dimensional building coordinates, and for the channel impulse response of each link, the noise stripping and baseline calibration are first completed:
[0108] Taking the reference response before the fire or the non-event period as the baseline, a fixed-length time window is constructed to difference the current response, and the early components of the transmitting end ringing and the near-field coupling are shielded; then multi-threshold peak detection and merging are performed on the remaining time delay sequence, the weak peak is retained according to the adjacent energy proportion, the number of repeated occurrences and the link geometric consistency, and the strong peak is retained after deduplication, and the center time delay and the cross-sample span are retained, to obtain the corresponding time delay component.
[0109] Each time delay component is combined with the positions of the double base points, the transmission and reception timing of the link and the calibrated clock bias to correspond to an isochronous delay elliptical trajectory in the building coordinate; to avoid excessive density of the trajectory near the boundary, an arc length is discretized with a fixed step, and the source link, peak intensity, detection confidence and time delay window width are recorded at each discrete point, and finally summarized as an elliptical ray diagram.
[0110] In the embodiment, the reference response and the current response of the link are differentially aligned in a window, and the window length and the sampling rate are determined by the device parameters, without limiting specific values; the peak detection uses a composite criterion based on neighborhood energy and minimum interval to avoid splitting the same reflection cluster into multiple false peaks; the clock calibration is completed by aligning the early direct components of a plurality of stable links in the layer, and the drift estimation result is shared in the entire layer to reduce systematic deviation across links; when the isochronous delay ellipse is discretized, a fixed arc length step is used, and the trajectory is truncated at the entity boundaries such as walls and floors, so that the trajectory only exists in the reachable space or the adjacent band outside the boundary.
[0111] Reference Figure 3 , Figure 3 The isochronous delay elliptical trajectory provided by the embodiment of the present application is shown in the figure.
[0112] Figure 3 A pair of ultra-wideband nodes in communication with each other are shown, and an isochronous delay elliptical trajectory calculated according to a time delay component in a channel impulse response covers an indoor area between the pair of ultra-wideband nodes, and is used to represent a set of potential reflection paths corresponding to the time delay component, wherein, Figure 3 The black dots in the figure represent entity obstacles where the objects to be positioned may exist, and the time delay component exists because of the entity obstacles.
[0113] It can be understood that, for the convenience of understanding, Figure 3 In the figure, the ultra-wideband nodes are drawn at the boundary positions of the indoor scene, and in actual applications, the ultra-wideband nodes are usually arranged at positions such as floor corridors, corners of walls or ceilings, so as to reduce the shielding of the indoor structure to the nodes while ensuring communication coverage.
[0114] S2.2: Map the elliptical ray diagram to a preset three-dimensional building information coordinate system, and assign an energy weight to a trajectory segment passing through a voxel to obtain an occupancy signal intensity of each voxel, and generate an initial occupancy grid map based on each voxel, wherein a grid in which the occupancy signal intensity corresponding to the voxel is greater than a preset voxel threshold is an occupancy grid, and the grid is set according to the building information;
[0115] Specifically, the whole floor space is discretized with fixed voxel size; voxel traversal calculation is then performed for each curve, using step-by-step voxel grid traversal method to count the effective path length and incident angle of the curve in each traversed voxel, and combining the peak intensity, detection confidence and time delay window width of the curve, the occupancy signal strength of the corresponding voxel is accumulated according to the preset weight distribution rule. For voxels that cross the entity boundary, if the voxel is not connected to any reachable space, only its geometric information as a barrier is recorded and does not participate in occupancy accumulation; for voxels near doorways or passages, the weight is amplified or suppressed according to the relationship between the doorway width and the curve incident angle, to reflect the difference in reachability. After all the curves are processed, the accumulated intensity of each voxel is density-normalized to eliminate the bias caused by the difference in the number of curve coverages in different areas, and the threshold is adaptively set according to the local sampling density and neighborhood variance, to generate an initial occupancy grid map, in which voxels with accumulated intensity higher than the threshold are marked as occupied grid.
[0116] In this embodiment, the voxel size is selected relative to the room scale and the minimum width of the passage, so that the doorway spans at least several voxel units to express the difference in reachability at the passage boundary; the weight distribution considers the curve peak intensity, detection confidence, path length, incident angle and time delay window width, the former two representing observation reliability, and the latter three representing geometric and temporal resolution; density normalization takes curve coverage count as the basis to prevent the intensity in the sensor-intensive area from being passively raised; threshold setting introduces local statistics to enable consistent occupancy determination in corridors, rooms and open areas under different sampling densities; for curves that cross the floor, layer-by-layer accumulation is performed in the three-dimensional voxel space, and narrow voxel layers are used in the floor neighborhood to record vertical traversal evidence, facilitating subsequent floor judgment and slice output.
[0117] Taking personnel positioning on a floor as an example, reference is made to Figure 4 , Figure 4 The initial occupancy grid map provided by the embodiments of the present application is shown in the schematic diagram.
[0118] Figure 4 It is shown that in the planar area of the floor, there are a total of 15 indoor rooms, each room is divided into several grids according to the rated area, and the initial occupancy grid result generated by the time delay component of the ultra-wideband node combined with the building information, in which the black squares represent grid units with occupancy signal strength exceeding the voxel threshold, representing the area where the target or obstacle may exist, and the remaining white squares represent the space not marked as occupied.
[0119] It can be understood that the size of the grid and the voxel threshold can be flexibly set according to the structural characteristics of different buildings and the deployment of sensors, and the present application does not make more limitations thereon.
[0120] S2.3: correcting the occupancy grid in the initial occupancy grid map according to the entity structure in the building information, to obtain an occupancy grid map;
[0121] Specifically, the initial occupancy grid map is checked item by item with the entity structure in the building information, the occupancy marks in the wall, floor or column voxels are removed, once occupancy rollback is performed in the normal direction of the structure boundary in the door frame and opening neighborhood, the high value falling inside the entity boundary is pushed back to the nearest reachable voxel along the normal direction; the occupancy bands at the channel center line fracture are bridged with the minimum gap criterion to ensure that a continuous reachable corridor is formed within the range allowed by the structure; in the areas such as equipment rooms and shafts where passing is not allowed, the occupancy high value is emptied according to the no-passing table; for isolated occupancy clusters that are not fully connected to the reachable space, the minimum connected domain area and the shortest reachable path are used as the deletion conditions to remove the local false peaks caused by individual link multipath.
[0122] In the embodiment, the normal rollback takes the normal of the wall and door opening boundary as the direction reference, and the rollback distance is limited by the door opening width and the accessibility of the adjacent voxels to avoid pushing the real wall-relying target away from the channel; the fracture bridging is triggered only when both ends are reachable voxels and the gap length does not exceed twice the minimum width of the channel to prevent unreasonable connectivity; the deletion threshold of the isolated cluster takes the floor area ratio and the historical time span as parameters, the clusters with small area and short time existence are preferentially deleted, and the clusters with long time stable existence are retained for multi-modal cross verification; for three-dimensional reconstruction, the correction operation is performed in the layer and between the layers of the voxel column respectively to ensure that the occupancy band of the vertically connected space remains continuous after slicing; after all the corrections are completed, a slight nonlinear compression is performed on the boundary neighborhood to pull the excessively high occupancy intensity back to the level consistent with the neighborhood, preventing excessive bias towards single source evidence in subsequent fusion.
[0123] Reference Figure 5 , Figure 5 The occupancy grid map schematic diagram provided by the embodiment of the application is shown in FIG. 1.
[0124] Figure 5 With Figure 4 With reference to the corresponding initial occupancy grid map, the initial occupancy result is corrected in combination with the entity structure in the building information, wherein the occupancy units in the unreachable areas such as walls and columns are removed, and the grid in the channel and door opening area is adjusted for connectivity, so that the occupancy result can more accurately reflect the real reachable space distribution inside the floor.
[0125] Next, the technical content of the water mist information about the method of the embodiment of the application is expanded.
[0126] It can be understood that the water mist information in the present application specifically refers to the water mist distribution intensity varying with space and time in the three-dimensional building information coordinate system. In the application scenario of the present application, the building usually includes various sprinkler devices, including but not limited to conventional sprinkler heads, water mist sprinkler heads, open sprinkler heads controlled by deluge valves, pre-action or dry system sprinkler heads, water curtain sprinkler heads, and local protection sprinkler heads, etc. It can also include temporary jet sources formed by fire hoses at fixed water hose interface positions.
[0127] The foregoing device is arranged in a partitioned manner in the floor and is connected to a wet, dry, pre-action, or deluge pipe network. The pipe network is usually provided with pressure gauges, flow switches, alarm valves, and regional valve position feedback signals, which are sufficient to provide operation data for calculating the working state of the sprinkler head and the water outlet condition.
[0128] As can be appreciated by those skilled in the art, the operation data can be obtained from building automation, fire pump room control cabinet, alarm valve group, or floor partition control box, etc. The specific interface form is not limited, as long as it can support the determination of whether the sprinkler head is opened, the water level and the duration at a minimum, and the present application does not make more limitations.
[0129] In the present embodiment, the calculation of water mist distribution intensity starts with partitioned sprinkler parameters, determines the opening time, duration, and water level for each possible water outlet source, and obtains the initial spray field in combination with the spray characteristic curve and installation attitude of the sprinkler head. The spray characteristic curve can be determined by the performance curve corresponding to the sprinkler head model and the geometric quantities such as cone angle, coverage radius, installation height, and distance from the wall. For open or water mist sprinkler heads, the relative water discharge intensity is converted according to the water supply pressure and water discharge coefficient. The sector-shaped coverage formed by the side wall type and upright type sprinkler head near the wall is processed according to the sector weight. When the deluge or water curtain forms linear coverage along the door opening, corridor or shaft boundary, the zonal distribution is generated in a line source manner. If the detailed model of a certain sprinkler head is missing, the conservative upper limit is set according to the general parameters of the same type sprinkler head on the same floor and the on-site calibration results, to ensure that the area with "heavy mist" can be correctly reflected in the case of insufficient estimation. The above-mentioned initial spray field is then superimposed with the building communication relationship of the floor: the flow field attenuation through the wall, the outward expansion in the direction of the door opening and the passage, and the shielding in the low ceiling or beam bottom are all corrected as spatial constraints; if the air supply and exhaust system is in operation, the spray field is diffused along the main airflow direction according to the fan working condition and the shaft position, to form a water mist distribution intensity more consistent with the fire field working condition.
[0130] Next, the technical content of the millimeter wave data preprocessing and thermal imaging data preprocessing of the embodiment method of the present application will be expanded.
[0131] It can be understood that in the multi-modal perception process in the fire scene, the data of the millimeter wave radar and the thermal imaging camera cannot be directly used for stable positioning, because the water mist and thermal disturbance in the environment will significantly interfere with the imaging principles of the two types of sensors.
[0132] The millimeter wave radar relies on electromagnetic waves to propagate in the environment and receive target reflection echoes. However, when there are a large number of suspended water droplets in the space, the electromagnetic waves will be scattered and attenuated multiple times between the water mist particles, causing the originally clear coherent echoes to have phase drift, group time delay deviation and energy dispersion, thereby causing deviations in the distance and angle estimation of the target. At the same time, volume scattering also forms random background noise clusters, making it difficult to distinguish human body echoes from water mist interference. If not pre-processed, the millimeter wave data may appear as false targets or trajectory drift on the three-dimensional imaging result, which is extremely easy to mislead the subsequent motion prior extraction.
[0133] The working principle of the thermal imaging camera is to form a temperature distribution map by detecting the infrared radiation of the surface of objects in the scene. However, in the fire sprinkling environment, the water mist particles will behave as a high-scattering and strong-absorbing medium in the infrared band, causing the thermal radiation signal to be disturbed before being detected by the camera. The direct manifestation is the appearance of obvious speckle flickering and brightness jitter in the thermal imaging picture, and even the generation of discontinuous occlusion patches in local areas, making boundary recognition difficult. Especially in the high-humidity and strong-convection fire field environment, thermal fluctuations will also superimpose airflow turbulence effects, making the time series of pixel points no longer stationary, and false edges and false high-temperature areas appear. If not pre-processed, the thermal imaging data may appear false human body contours in the boundary extraction link, destroying the reliability of multi-modal fusion.
[0134] In one example, the pre-processing of the millimeter wave data according to the water mist information includes:
[0135] S301: According to the building information, a static reference reflector in the corresponding floor is selected as a calibration target;
[0136] Specifically, the reference reflector is used to provide stable phase and time delay anchors under the conditions of water mist and airflow disturbance, so as to eliminate global drift caused by temperature and humidity changes, sprinkler starting and device self-heating. Through the building information model, it can be known that the components in each floor that do not move long-term and have stable mirror reflection characteristics to millimeter waves, such as metal door frames, stair handrail columns, power distribution cabinet covers, equipment room door leaves, and corridor corner steel components, etc. These components appear as high signal-to-noise, fixed position, and stable spectral peak shape echoes in the field of view.
[0137] S302: Combining the distance-angle-velocity data of each frame in the millimeter wave data, the common phase error and group time delay deviation corresponding to the calibration target are calculated;
[0138] Specifically, the calibration targets are fixed in position and have stable line of sight in the floor, which can be used as anchor points for coherent calibration of each frame. According to the building information and the external parameters of the radar installation, the distance unit and angle beam of each calibration target are first predicted, and then the complex echo vector in the zero or near-zero velocity gate is extracted from the distance-angle-velocity data cube in the neighborhood of the calibration target in each frame. The main lobe position is locked through a small range peak search, and the complex amplitude, phase and corresponding distance unit index of the position are recorded. The above measured quantities are compared with the reference quantities obtained from the geometric path: the phase difference between the two is the common phase error of the calibration target in the current frame, and the distance unit difference and the fast time correlation peak shift around the point jointly depict the group delay deviation of the calibration target.
[0139] In this embodiment, in order to make the common item estimation insensitive to occlusion and local disturbance, a robust weighted aggregation method for multiple calibration targets is adopted: the weight of each calibration target is determined by the signal-to-noise ratio, the angle error with the predicted beam, the water mist distribution intensity and the visibility of the point in the recent time window; observations with too low weight or too large deviation from the group are automatically excluded. The aggregation of phase offset uses ring mean method to avoid cross-week variation, and the aggregation of time delay offset obtains the peak shift estimation of sub-sampling level through fast time correlation with the reference frame, and then does exponential smoothing to suppress transient disturbance. In order to avoid over extrapolation of the common item between different sectors of the field of view, the common phase and group delay are first estimated in angular sectors in the frame, and then fused into the full-frame common value with sector coverage as weight; when the number of effective calibration targets is less than the set lower limit, the last common item is frozen and only local correction is applied in the sector with calibration targets to ensure the continuity of the output.
[0140] S303: calculating a phase structure function according to the water mist information, the common phase error and the group delay deviation;
[0141] Specifically, the phase structure function is used to describe the correlation and scale of phase disturbance in each direction in space, and is the basis for subsequent interpolation and compensation. The higher the water mist distribution intensity and the more turbulent the airflow, the shorter the phase correlation length and the worse the coherence between beams; encoding this rule into the structure function can make interpolation only occur within a reliable spatial scale.
[0142] In this embodiment, according to the water mist distribution intensity field, the floor is divided into several working condition areas, each area is given a set of phase disturbance parameters, including spatial correlation length, time update rate and amplitude upper limit; at the same time, the common phase error and the group delay deviation introduced as zero bias constraints make the phase structure function globally anchored to the reference body. The phase structure function takes angle and distance as the main axis, allowing a larger correlation length in the long direction of the corridor and a smaller correlation length in the normal direction of the wall, so as to fit the propagation characteristics of the corridor type scene.
[0143] S304: interpolating the millimeter wave data according to the phase structure function to obtain corrected data, wherein the interpolation is used to restore the coherence of the millimeter wave data;
[0144] Specifically, the target of the interpolation is to resample and phase unwrap the observation with slight distance walk caused by the water mist disturbance, to restore the coherence of the cross-beam and cross-range unit, by using the spatial correlation given by the phase structure function under the premise that the known common drift is removed.
[0145] In some optional specific embodiments, the complex echo in each angular sector is first globally unwrapped according to the common phase and group delay, so that the phase and distance at each reference body are aligned with the baseline; then in the local neighborhood of the angle and distance, the shape and weight of the interpolation kernel are determined according to the phase structure function, and the missing or mismatched complex samples are supplemented and smoothed; for the samples falling into the strong water mist area, the maximum phase unwrapping amount and the interpolation step are limited to prevent false coherence caused by excessive extrapolation. After completing the local interpolation, the consistency of the multi-frame samples of each beam is checked, and the isolated modified points that are not continuous in adjacent time are removed to form the corrected data.
[0146] S305: transforming the corrected data to the beam domain by a subspace decomposition algorithm, and purifying the corrected data in the beam domain by a preset decomposition model to obtain purified millimeter wave beam domain data, wherein the decomposition model includes a low-rank background term and a sparse target term, the low-rank background term corresponds to water mist volume scattering and non-directional echo, and the sparse target term corresponds to human body reflection echo;
[0147] Specifically, the water mist volume scattering is a diffuse background with a large range and slow change in space, and the human body reflection is a sparse feature with local and prominent characteristics in space and speed. After transforming the corrected data to the beam domain, low-rank-sparse decomposition can be used to separate the background and the target, and then purify the data.
[0148] In this embodiment, the preset of the decomposition model is based on modeling the statistical characteristics of the beam domain matrix. Taking the beam-range matrix of each frame as the basic unit, the time dimension or the velocity dimension is stacked as an additional channel to obtain a three-order tensor matrix. In the three-order tensor matrix, the non-directional echo caused by the water mist volume scattering has similar energy distribution on most beams and multiple range cells, and has strong time continuity, so it has the characteristics of low rank value and can be described by a few principal components; and the human body reflection point only appears in a few beams and local range cells, and the velocity component is concentrated in a limited interval, so it appears as an isolated and discontinuous protrusion in the matrix, which can be highlighted by sparse constraint.
[0149] In the preset stage, the upper limit of the rank of the low-rank background term and the sparsity threshold of the sparse target term need to be defined. The setting of the upper limit of the rank is combined with the water mist distribution intensity and the wind field state. When the water mist intensity is high, a higher rank value is allowed to retain the slowly changing scattering mode. When the water mist is lower, the upper limit of the rank is tightened to avoid the background from absorbing too much target information. The sparsity threshold is adjusted according to the typical radar cross section range, motion speed interval and scattering characteristics of the human body, so that only local sharp energy can enter the sparse term, and the part with continuous distribution and slowly varying energy is pressed into the low-rank background.
[0150] Further, the purification process is realized by alternating iteration: first, singular value decomposition is used to decompose the beam domain matrix, the first several singular values and the corresponding vectors are intercepted to reconstruct the low-rank background term, and the remaining part is used as the candidate sparse term; then, the sparsity constraint is applied to the candidate sparse term, the local energy peak exceeding the threshold is retained and other noise is suppressed; then the background term and the target term are combined into a new matrix, compared with the original observation, if the residual error is less than the threshold, the process is stopped, otherwise the iteration is continued. Through this process, the low-rank background term gradually converges to the water mist scattering mode covering the whole field, and the sparse target term retains the local human body reflection peak.
[0151] S306: The purified millimeter wave beam domain data is de-projected to obtain pre-processed millimeter wave data;
[0152] In one example, the thermal imaging data is pre-processed according to the water mist information, including:
[0153] S311: According to the arrangement position corresponding to the water mist distribution intensity, the spatial gradient of the water mist distribution intensity is calculated, and the local flow field of the spatial gradient is calculated in combination with the position of the thermal imaging camera;
[0154] Specifically, the water mist distribution intensity has been aligned with the floor grid in the three-dimensional coordinate system. In order to obtain the flow direction information required for thermal imaging de-disturbance, the change direction of the intensity in the plane needs to be calculated and related to the imaging geometry of the thermal imaging camera. The spatial gradient is used to describe the rising direction from the low-intensity area to the high-intensity area. In combination with the building connectivity and the supply and exhaust boundary conditions, the overall migration trend of the mist droplets can be inferred.
[0155] In this embodiment, the water mist distribution intensity is calculated by a structure-guided difference method to obtain the spatial gradient: a longer difference step is allowed in the longitudinal direction of the channel, and only the inside single difference is taken in the wall normal direction to avoid direction reversal caused by crossing the entity boundary; a correction term is introduced in the door opening neighborhood to ensure that the gradient can be continuous across the area. The obtained gradient vector is smoothed by neighborhood consistency, the corridor direction retains the slender structure, and the corner is limited to prevent excessive smoothing to retain the inflection point. The internal and external parameters of the thermal imaging camera are calibrated during deployment, the projection of the pixel to the floor plane adopts the plane intersection relationship, the falling point of each pixel is queried in the grid unit, and the corresponding gradient vector is obtained by interpolation. For pixels located in high-level overhead areas or behind shielding bodies, mark them as low-confidence areas, and only participate in time-domain baseline estimation in the future, not in directional filtering.
[0156] S312: Calculate the flow line direction of each pixel in the thermal imaging data according to the principal component flow line of the local flow field;
[0157] Specifically, the local flow field may have fine-grained disturbances and local eddies on the image plane, and it is necessary to determine a stable dominant flow line direction for each pixel, which is consistent with the neighborhood and avoids direction jumps at boundaries and corners. The determination idea of the principal component flow line direction is: sample the local flow vectors of multiple points in the pixel neighborhood, select the dominant direction that can best explain the direction distribution of the neighborhood, and ensure that the direction changes smoothly with space.
[0158] S313: Extract the time series of each pixel in the flow line direction, and perform time-domain filtering processing on the time series, wherein the filtering window length is calculated according to the water mist distribution intensity of the corresponding position;
[0159] Specifically, the particle flicker and brightness jitter caused by water mist in thermal imaging present short-period and unstable disturbances in the time dimension. Extracting and filtering the time series of the pixel along the real migration direction can separate the moving particles caused by the movement of the droplet group from the stable contour of the human body. The filtering window length and weight need to be adaptively adjusted according to the water mist distribution intensity. The thicker the fog, the longer the window and the stronger the smoothing; when the fog is weak, more boundary details are retained.
[0160] S314: According to the time series after time-domain filtering processing, calculate the flicker frequency band through frequency spectrum analysis, perform suppression processing on the flicker frequency band, and obtain the preprocessed thermal imaging data by summarizing the time series after suppression processing of each pixel, wherein the suppression processing reduces the energy amplitude of the flicker frequency band through a notch filter, and the parameters of the notch filter are calculated according to the water mist distribution intensity of the corresponding position;
[0161] Specifically, the modulation of flame on infrared is usually concentrated in a limited frequency band, and the brightness fluctuation formed when water mist particles are carried by airflow through the field of view is also accumulated near this frequency band. If this frequency band can be identified at the pixel level or small block level and its energy is adaptively reduced, the flicker can be suppressed without sacrificing the edge transition. The notch parameters need to be determined according to the water mist intensity and time residual, so that the suppression range and depth are appropriately matched to the local disturbance scale.
[0162] In this embodiment, a fixed-length short-time analysis window is used for spectral evaluation of the time series output by S313, and the window length is determined by the camera frame rate and the target frequency band coverage requirement. A cosine weighting is used to reduce the boundary effect; the power spectrum is aggregated in the pixel neighborhood to find the main peak and the secondary peak in the target frequency band, and the main peak frequency is used as the notch center. The bandwidth is given by the spectral peak half-width and the water mist intensity, and the intensity is high, the bandwidth and the suppression depth are increased, and the intensity is low, the bandwidth is tightened; when there is no obvious peak value in the spectrum and the time domain residual is still high, a wideband shallow suppression strategy is enabled to prevent overfitting. The notch filter uses a phase-preserving design to ensure that the low-frequency components of the boundary motion are not affected.
[0163] Next, the technical content of the method of the embodiment of the application on millimeter wave data processing and thermal imaging data processing is expanded.
[0164] In one example, the motion prior information is calculated according to the preprocessed millimeter wave data, including:
[0165] S401: Distance-angle-velocity three-dimensional imaging is performed on the preprocessed millimeter wave data to obtain a three-dimensional data cube;
[0166] S402: A falling speed suppression band associated with water mist information is set in the velocity dimension, and the three-dimensional data cube is detected according to the falling speed suppression band to obtain a motion candidate unit;
[0167] S403: The motion candidate unit is clustered in the distance-angle two-dimensional plane to obtain a target scattering cluster;
[0168] S404: For each target scattering cluster, the radial velocity, azimuth angle and distance parameters are calculated, and the velocity vector and the confidence degree corresponding to the velocity vector of the target scattering cluster are calculated through multi-target trajectory estimation combining adjacent frames, and the velocity vector and the corresponding confidence degree are taken as the motion prior information.
[0169] In one example, the boundary constraint information is calculated according to the preprocessed thermal imaging data, including:
[0170] S411: Multi-scale gradient detection and region growing are performed on the preprocessed thermal imaging data to extract temperature mutation edges to obtain a candidate boundary contour;
[0171] S412: filtering the candidate boundary contour according to the preset temperature threshold and contrast threshold to obtain a contour region conforming to human radiation characteristics;
[0172] S403: performing morphological processing and connected domain analysis on the contour region to generate a closed boundary curve, and generating boundary constraint information according to the closed boundary curve.
[0173] Next, the technical content of the embodiment method of the present application on the fusion solving of the occupancy grid is expanded.
[0174] It can be understood that the fusion solving of the occupancy grid takes three types of data aligned with the floor coordinates as input: the occupancy grid corrected by S2.3, the motion prior obtained by millimeter wave preprocessing and rasterized, and the closed boundary constraint obtained by thermal imaging preprocessing and projected to the floor plane. The general idea of fusion is to regard the occupancy intensity as evidence that can be transferred in time, the velocity prior as the flow direction of evidence, and the closed boundary as the container and limit of evidence; the three jointly constrain the evolution of evidence in the grid, so that the high-value area is aggregated along the reachable path and attenuated in the unreachable area. This processing avoids phenomena such as wall jumping and drifting caused by single-source jitter under strong water mist and shielding conditions, and makes the continuous evidence in time and the connected relationship in space consistent.
[0175] In one example, the motion prior information and the boundary constraint information are fused and solved in the occupancy grid, including:
[0176] S5.1: taking the projection of the boundary constraint information in the occupancy grid as a boundary mask, and taking intersection with the occupancy grid to obtain an occupancy density map;
[0177] Specifically, the closed boundary curve is derived from the contour after thermal imaging preprocessing, and its geometric reference is in the floor plane coordinates, which is not naturally the same resolution and phase as the discrete grid of the occupancy grid. Directly using the boundary in subsequent calculations will cause problems such as inconsistent boundary thickness, disconnected at doorways, and missing labels caused by the tangency of the boundary and the grid line. In order to have a definite expression of the envelopable area on the discrete grid, it is necessary to stably rasterize the boundary curve into a binary mask, and pre-slot at doorways and passages according to the architectural connectivity, so that the subsequent evidence can not only accumulate within the boundary, but also transfer across regions along the real reachable passage, thereby avoiding the destruction of reachability expression caused by geometric discretization errors.
[0178] In this embodiment, the boundary curve is projected at sub-pixel level by multi-segment line scanning conversion, and the curve is resampled at a fixed step to be a short line segment and then written into a grid. To suppress the holes caused by the excessively fine boundary, the curve is first executed a one-cell scale dilation, and then a one-sided erosion is performed on one side of the wall to obtain a boundary band with consistent thickness and falling on the accessible side. The door hole position is given by the building information, and a pass-through slot is opened on the mask with the inside of the door frame as the reference, the slot width is consistent with the net width of the door hole, and the slot is extended by one or two grids at both ends according to the direction of the passage center line to ensure the cross-region continuity in the advection stage. Then, the boundary mask is intersected with the occupancy grid modified by S2.3 to obtain an occupancy density map, wherein the non-zero cells represent the regions that satisfy the occupancy evidence and the boundary envelope at the same time, and the regions outside the boundary or blocked by the entity are automatically emptied.
[0179] S5.2: performing voxel-level quality-conserved advection updating on the occupancy density map according to the motion prior information to obtain an evidence density corresponding to the occupancy grid, wherein the quality-conserved advection updating calculates an updated value of the occupancy grid through a backtracking sampling rule along a velocity direction, and the backtracking sampling rule includes inhibiting flow in the wall and the unconnected region and enhancing flow at the door hole and the passage connection;
[0180] Specifically, the occupancy density map represents the static evidence distribution at the moment, and the motion prior provides the direction and strength of the evidence transfer. If the advection is not described by the quality conservation, the evidence will produce ungrounded diffusion or disappear in the air in the numerical value. By using the backtracking sampling along the velocity direction, the current value of each cell can be traced back to where it is generated in the last step, so that the conservation transport of the evidence is realized on the discrete time step; at the same time, the transport is made to comply with the accessibility and the pass preference by combining the wall inhibition and the door hole enhancement.
[0181] In this embodiment, the velocity vector and the confidence of each non-zero cell are read, and the position of the last step is traced back at a fixed time step; the backtracking path uses a line segment crossing judgment and an entity mask query, if the path crosses the wall or the closed boundary, the backtracking weight of the cell is set to zero; if the path passes through the door hole slot and is close to the passage center line, the weight is linearly gained according to the distance from the center line. The source density of the backtracking position is obtained by bilinear interpolation, and is multiplied by the velocity confidence as the evidence inflow transported to the cell. To avoid numerical oscillation, a small anisotropic viscous term is added to each cell, which only weakly smooths along the velocity direction and does not diffuse along the normal direction.
[0182] S5.3: normalizing and synthesizing the evidence density and the occupancy grid to obtain an updated occupancy grid, wherein the synthesis weight of the normalizing and synthesizing is calculated according to the velocity confidence and the boundary consistency;
[0183] Specifically, the advection obtains time-series evidence dominated by motion, while the occupancy grid modified by S2.3 preserves evidence of consistency with multi-link latency and structural priors. Both are consistent in some areas and contradictory in others. Taking one directly will cause the loss of information of the other. The speed confidence and the boundary consistency are used as the basis for the normalization and synthesis of the two routes of evidence, so that continuous peak bands can be formed along the motion direction, and the spatial range defined by the structure and the thermal boundary is not deviated.
[0184] In this embodiment, two types of weights are calculated for each cell: the speed weight is given by the stability of the velocity vector, the spectral peak sharpness, and the millimeter wave pre-processing residual, and the boundary weight is given by the shortest distance from the cell to the boundary, the boundary definition, and the boundary consistency index of S5.1; then the evidence density and the basic occupancy are synthesized with these two types of weights, and the synthesized value is limited within the neighborhood statistics range to suppress mutations.
[0185] S5.4: selecting a plurality of peak regions in the updated occupancy grid map, calculating the centroid of each peak region as the position information of the object to be positioned, and outputting the principal axis direction of each peak region as the motion direction;
[0186] Specifically, the high-value region in the updated occupancy grid map corresponds to the candidate target, but the high value may be an extended band of a single target or a superposition of adjacent multiple targets. Directly taking the maximum value will cause unstable positioning and target sticking, so it is necessary to determine the true peak area through spatial suppression and connected domain analysis, and then output the position and direction with a statistically meaningful physical quantity, so that the result is not sensitive to local noise and can maintain separability when multiple targets coexist.
[0187] In one example, the embodiments of the present application provide an indoor personnel positioning system based on multi-modal perception, the system comprising:
[0188] A perception acquisition module is configured to acquire channel impulse responses of ultra-wideband links, millimeter wave data of millimeter wave radars, and thermal imaging data of thermal imaging cameras on each floor of a multi-story building, and receive building information and spraying parameters of a spraying device to calculate a water mist distribution intensity;
[0189] A grid reconstruction module is configured to perform double-base geometric mapping according to the channel impulse responses and the building information, reconstruct indoor elliptical ray paths, and generate an occupancy grid map; and correct the millimeter wave data and beam domain purification according to the water mist distribution intensity, perform time domain filtering and flicker band suppression on the thermal imaging data along the flow direction, and output pre-processed millimeter wave data and thermal imaging data;
[0190] A constraint extraction module is configured to calculate motion prior information from the pre-processed millimeter wave data; and extract a closed boundary curve from the pre-processed thermal imaging data to generate boundary constraint information;
[0191] The fusion positioning module is configured to fuse and solve the motion prior information and the boundary constraint information in the occupancy grid map, including voxel-level mass conservation advection updating and normalized synthesis, and determining the center of mass and the principal axis of the peak region in the updated occupancy grid map, and outputting the position information and the motion direction of the object to be positioned.
[0192] The constraint extraction module comprises:
[0193] The motion prior unit is configured to perform three-dimensional imaging and connected domain clustering on the preprocessed millimeter wave data, calculate the radial velocity, azimuth and distance parameters of the target scattering cluster, and output the velocity vector in combination with adjacent frame trajectory estimation, to generate the motion prior information;
[0194] The boundary generation unit is configured to perform multi-scale edge detection and region growing on the preprocessed thermal imaging data, screen out the contour region meeting the human body radiation characteristics, and generate a closed boundary curve through morphological processing and connected domain analysis, to obtain the boundary constraint information.
[0195] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.
Claims
1. A multi-modal perception based indoor personnel positioning method, applied to indoor positioning of an object to be positioned in a multi-story building, characterized in that, The method comprises: acquiring channel impulse responses, millimeter wave data and thermal imaging data of each floor; reconstructing an indoor elliptical ray path by double-baseline geometric mapping according to the channel impulse responses and combining preset building information to obtain an occupancy grid map; collecting water mist information, preprocessing the millimeter wave data and thermal imaging data according to the water mist information, calculating motion prior information according to the preprocessed millimeter wave data, and calculating boundary constraint information according to the preprocessed thermal imaging data; fusing and solving the motion prior information and the boundary constraint information in the occupancy grid map to obtain position information of an object to be positioned.
2. The multi-modal perception based indoor personnel positioning method according to claim 1, wherein, The method comprises: taking a transmitting end and a receiving end of an ultra-wideband node in communication as a pair of double-baseline points, calculating isochronous delay elliptical trajectories corresponding to the pair of double-baseline points according to propagation times of corresponding delay components in the channel impulse responses, and obtaining an elliptical ray map by summarizing the isochronous delay elliptical trajectories; mapping the elliptical ray map to a preset three-dimensional building information coordinate system, assigning energy weights to trajectory segments crossing voxels to obtain occupancy signal strengths of the voxels, and generating an initial occupancy grid map taking the voxels as references, wherein an occupancy grid in which an occupancy signal strength corresponding to a voxel is greater than a preset voxel threshold is an occupancy grid, and the grid is set according to building information; correcting the occupancy grid in the initial occupancy grid map according to entity structures in the building information to obtain an occupancy grid map.
3. The multi-modal perception based indoor person localization method of claim 1, wherein, The water mist information is water mist distribution intensity, and the water mist distribution intensity is collected in the following manner: acquiring spraying parameters of spraying equipment on each floor in building information, wherein the spraying parameters include an opening state of a spray head, a spraying flow rate and a spraying pressure; calculating water mist distribution intensities of each floor according to the spraying parameters, combining arrangement positions of the spraying equipment and a preset spraying characteristic curve.
4. The method of claim 3, wherein, The preprocessing of the millimeter wave data according to the water mist information comprises: selecting a static reference reflector in a corresponding floor as a calibration target according to building information; calculating a common phase error and a group delay deviation corresponding to the calibration target in combination with distance-angle-velocity data of each frame in the millimeter wave data; calculating a phase structure function according to the water mist information, the common phase error and the group delay deviation; performing interpolation processing on the millimeter wave data according to the phase structure function to obtain corrected data, wherein the interpolation processing is used to restore coherence of the millimeter wave data; transforming the corrected data to a beam domain by a subspace decomposition algorithm, performing purification processing on the corrected data in the beam domain by a preset decomposition model to obtain purified millimeter wave beam domain data, wherein the decomposition model includes a low-rank background term and a sparse target term, the low-rank background term corresponds to water mist volume scattering and non-directional echoes, and the sparse target term corresponds to human body reflection echoes. The purified millimeter wave beam domain data is de-projected to obtain pre-processed millimeter wave data.
5. The multi-modal perception based indoor person localization method of claim 3, wherein, The thermal imaging data is pre-processed according to the water mist information, including: According to the arrangement position corresponding to the water mist distribution intensity, the spatial gradient of the water mist distribution intensity is calculated, and the local flow direction field of the spatial gradient is calculated in combination with the position of the thermal imaging camera; According to the principal component flow line of the local flow direction field, the flow line direction of each pixel in the thermal imaging data is calculated; In the flow line direction, the time sequence of each pixel is extracted, and the time sequence is processed by time domain filtering, wherein the filtering window length is calculated according to the water mist distribution intensity of the corresponding position; According to the time sequence processed by the time domain filtering, the flicker frequency band is calculated by frequency spectrum analysis, the flicker frequency band is suppressed, and the time sequence after the suppression processing of each pixel is summarized to obtain pre-processed thermal imaging data, wherein the suppression processing reduces the energy amplitude of the flicker frequency band by a notch filter, and the parameters of the notch filter are calculated according to the water mist distribution intensity of the corresponding position.
6. The multi-modal perception based indoor person localization method of claim 1, wherein, The motion prior information is calculated according to the pre-processed millimeter wave data, including: The pre-processed millimeter wave data is subjected to distance-angle-velocity three-dimensional imaging to obtain a three-dimensional data cube; A falling speed suppression band associated with the water mist information is set in the velocity dimension, and the three-dimensional data cube is detected according to the falling speed suppression band to obtain a motion candidate unit; The motion candidate unit is subjected to connected domain clustering in the distance-angle two-dimensional plane to obtain a target scattering cluster; For each target scattering cluster, the radial velocity, azimuth angle and distance parameters are calculated, and in combination with adjacent frames, the velocity vector of the target scattering cluster and the confidence degree corresponding to the velocity vector are calculated by multi-target trajectory estimation, and the velocity vector and the corresponding confidence degree are taken as the motion prior information.
7. The multi-modal perception based indoor person localization method of claim 1, wherein, The boundary constraint information is calculated according to the pre-processed thermal imaging data, including: Multi-scale gradient detection and region growing are performed on the pre-processed thermal imaging data to extract a temperature mutation edge to obtain a candidate boundary contour; The candidate boundary contour is screened according to a preset temperature threshold and a contrast threshold to obtain a contour region meeting human body radiation characteristics; The contour region is subjected to morphological processing and connected domain analysis to generate a closed boundary curve, and the boundary constraint information is generated according to the closed boundary curve.
8. The multi-modal perception based indoor person localization method of claim 1, wherein, The motion prior information and the boundary constraint information are fused and solved in the occupancy grid map, including: The projection of the boundary constraint information in the occupancy grid map is taken as a boundary mask, and the occupancy density map is obtained by taking intersection with the occupancy grid; The occupancy density map is subjected to voxel-level mass conservation advection update according to the motion prior information to obtain the evidence density corresponding to the occupancy grid, wherein the mass conservation advection update calculates the update value of the occupancy grid by a backtracking sampling rule along the velocity direction, and the backtracking sampling rule includes inhibiting the flow of wall bodies and non-connected regions and enhancing the flow at door hole and channel communication positions. The evidence density is normalized and synthesized with the occupancy grid map to obtain an updated occupancy grid map, wherein a synthesis weight of the normalized synthesis is calculated according to a speed confidence and a boundary consistency; A plurality of peak regions are selected in the updated occupancy grid map, a centroid of each peak region is calculated as position information of the object to be positioned, and a principal axis direction of each peak region is output as a motion direction.
9. A multi-modal perception based indoor people positioning system for implementing a multi-modal perception based indoor people positioning method according to any one of claims 1-8, characterized in that, The system comprises: An awareness acquisition module is configured to acquire a channel impulse response of an ultra-wideband link, millimeter wave data of a millimeter wave radar, and thermal imaging data of a thermal imaging camera on each floor of a multi-story building, and receive building information and spraying parameters of a spraying device to calculate a water mist distribution intensity; A grid reconstruction module is configured to perform double-base geometric mapping according to the channel impulse response and the building information, reconstruct an indoor elliptical ray path, and generate an occupancy grid map; and correct the millimeter wave data and beam domain purification according to the water mist distribution intensity, perform time domain filtering along a flow direction and flicker band suppression on the thermal imaging data, and output preprocessed millimeter wave data and thermal imaging data; A constraint extraction module is configured to calculate motion prior information from the preprocessed millimeter wave data, and extract a closed boundary curve from the preprocessed thermal imaging data to generate boundary constraint information; A fusion positioning module is configured to fuse and solve the motion prior information and the boundary constraint information in the occupancy grid map, including voxel-level mass conservation advection updating and normalized synthesis, and determining a centroid and a principal axis of a peak region in an updated occupancy grid map to output position information and a motion direction of an object to be positioned.
10. The multi-modal perception based indoor personnel positioning system according to claim 9, wherein, The constraint extraction module comprises: A motion prior unit is configured to perform three-dimensional imaging and connected domain clustering on the preprocessed millimeter wave data, calculate radial velocity, azimuth angle, and distance parameters of a target scattering cluster, and output a velocity vector in combination with adjacent frame trajectory estimation to generate motion prior information; A boundary generation unit is configured to perform multi-scale edge detection and region growing on the preprocessed thermal imaging data, screen out a contour region conforming to human radiation characteristics, and generate a closed boundary curve through morphological processing and connected domain analysis to obtain boundary constraint information.
Citation Information
Patent Citations
Multi-unmanned aerial vehicle cooperative positioning-oriented UWB and vision fusion positioning method and system
CN117685953A
Multi-mode fusion robust sensing positioning method and system for underground severe environment
CN118031931A
