A method and system for building data processing and visual generation

Abstract

The application discloses a kind of building data processing and visual generation method and system, it is related to building management and intelligent building technical field, method includes the following steps: acquisition building sensor data and visual data, sensor data is carried out time series preprocessing, abnormality detection and generates sensor confidence measure, visual data is carried out feature extraction and mapping to sensor reference space generates visual confidence measure;Sensor confidence measure generated and mapped visual confidence measure are as input, construct multimodal joint confidence alignment target, and determine learnable mapping function, simultaneously introduce time smoothing constraint to ensure the continuity of multimodal data on time series.In the application, by multimodal acquisition, feature fusion to building sensor data and visual data is generated synthetic confidence measure and is used for visual rendering, to improve the problem that traditional building management uses single data source, because it cannot be fused multimodal information causes state monitoring to be not complete.

Description

Technical Field

[0001] This invention relates to the field of building management and intelligent building technology, and in particular to a method and system for building data processing and visualization generation. Background Technology

[0002] With the rapid development of smart cities and intelligent buildings, building management systems are evolving towards digitalization and intelligence. Modern buildings deploy various types of sensors to monitor temperature, humidity, energy consumption, safety, and equipment status in real time, while also widely employing visual acquisition devices to obtain spatial layout and environmental information. The collection of multimodal data can provide more comprehensive decision-making support for building management; however, due to the diverse data sources, inconsistent formats, and complex temporal and spatial distribution, traditional building management systems struggle to achieve unified processing and fusion of multimodal data.

[0003] Currently, traditional building management mostly uses a single data source, which cannot integrate multimodal information, resulting in incomplete building status monitoring. Summary of the Invention

[0004] To overcome the above shortcomings, this invention provides a method and system for building data processing and visualization generation, aiming to improve the problem of incomplete status monitoring caused by the inability to integrate multimodal information in traditional building management which uses a single data source.

[0005] In a first aspect, the present invention provides the following technical solution: a method for building data processing and visualization generation, comprising the following steps:

[0006] S1. Collect building sensor data and visual data, perform time series preprocessing and anomaly detection on the sensor data and generate sensor confidence measures, and extract features from the visual data and map them to the sensor reference space to generate visual confidence measures.

[0007] S2. Using the generated sensor confidence measure and the mapped visual confidence measure as input, construct a multimodal joint confidence alignment target and determine a learnable mapping function. At the same time, introduce a time smoothing constraint to ensure the continuity of multimodal data in the time series.

[0008] S3. Iteratively solve the synthetic confidence measure and mapping function parameters by alternating minimization strategies until the joint optimization objective converges, and update the synthetic confidence measure in each iteration to fuse visual and sensor information;

[0009] S4. The converged synthetic confidence measure is transformed into a structured context and input into a large language model to generate a visualization script to constrain the visualization elements to be consistent with the confidence of the multimodal data.

[0010] S5. Render two-dimensional or three-dimensional visualization layers of the building based on the generated visualization script, mark the spatial location, status information and confidence level, and form a visualization output that can be directly used for building management and monitoring.

[0011] By adopting the above technical solution, multimodal acquisition of building sensor data and visual data is performed, feature fusion is used to generate synthetic confidence measures and used for visualization rendering, thereby improving the problem of incomplete status monitoring caused by the inability to integrate multimodal information in traditional building management which uses a single data source.

[0012] Preferably, the generation of sensor confidence measures includes:

[0013] Linear or polynomial interpolation is performed on the collected building sensor time series data to fill in missing data;

[0014] Statistical anomaly detection is applied to the interpolated data, including mean-variance determination and sliding window detection, to remove outlier sampling points;

[0015] The confidence value of each sensor is calculated based on the stability of historical sensor data and the fluctuation of real-time response.

[0016] The processed sensor data and corresponding confidence values ​​are standardized and time-aligned to generate a sensor confidence measurement data matrix in a unified format.

[0017] Preferably, the generation of the visual confidence measure includes:

[0018] Feature extraction is performed on the collected building visual data using convolutional neural networks or visual language models, including edges, structural textures, and semantic tags;

[0019] Based on the building space topology or CAD model information, visual features are mapped to the sensor reference coordinate system through affine or projection transformations.

[0020] Calculate the confidence score for the mapped visual features to generate a visual confidence measure;

[0021] The visual confidence measure is processed by time series analysis and spatial grid division to form a data matrix that can be directly used for joint alignment.

[0022] Preferably, the objective for constructing the multimodal joint confidence alignment includes:

[0023] Using the sensor confidence measure matrix and the visual confidence measure matrix as input, a joint confidence alignment objective function is constructed.

[0024] Apply spatial consistency constraints to the objective function to ensure spatial alignment between the sensor and visual data;

[0025] Apply statistical consistency constraints to ensure that the fused data maintains a consistent confidence distribution;

[0026] Apply regularization constraints to the mapping function to ensure spatial continuity;

[0027] Apply a sliding window smoothing constraint to the time series data to ensure time continuity.

[0028] Preferably, the parameters of the synthetic confidence measure and mapping function include:

[0029] The synthesized confidence measure is initialized as a weighted average of the sensor confidence measures, and the mapping function parameters are initialized as unit mapping or random perturbation;

[0030] With fixed mapping function parameters, the synthetic confidence measure is updated by minimizing the joint confidence alignment objective.

[0031] With a fixed synthetic confidence measure, update the mapping function parameters using gradient descent or alternating minimization strategies;

[0032] Repeat the steps until the joint objective function converges or the upper limit of iteration is reached;

[0033] Output the converged synthetic confidence measure matrix to provide a data foundation for visualization generation.

[0034] Preferably, the generation of the visualization script includes:

[0035] The converged synthetic confidence measure matrix is ​​transformed into a structured context, including node locations, state attributes, and confidence information.

[0036] Input structured context into a large language model, and generate visualization scripts through templates or instructions to make the spatial position and confidence of visualization elements correspond;

[0037] The generated script is parsed to map two-dimensional or three-dimensional graphic elements to the building coordinate system.

[0038] Preferably, the rendered building two-dimensional or three-dimensional visualization layer includes:

[0039] Receive element information generated by a structured context or a visualization script;

[0040] Map node information to a two-dimensional or three-dimensional coordinate system of the building;

[0041] Render visual elements, including node position, shape, and color encoding;

[0042] Generate legends or labels based on node attributes and confidence levels;

[0043] Output complete visualization layers or datasets for building management or monitoring.

[0044] Secondly, the present invention provides the following technical solution: a system for building data processing and visualization generation, comprising the following modules:

[0045] The data acquisition and preprocessing module is used to acquire building sensor data and visual data, perform time series preprocessing and anomaly detection on sensor data and generate sensor confidence measures, and extract features from visual data and map them to sensor reference space to generate visual confidence measures.

[0046] The multimodal joint alignment module is used to construct a multimodal joint confidence alignment target by taking the generated sensor confidence measure and the mapped visual confidence measure as input, and to determine the learnable mapping function. At the same time, a time smoothing constraint is introduced to ensure the continuity of multimodal data in the time series.

[0047] The iterative solution module is used to iteratively solve the synthetic confidence measure and mapping function parameters by alternating minimization strategies until the joint optimization objective converges, and updates the synthetic confidence measure in each iteration to fuse visual and sensor information;

[0048] The visualization script generation module is used to transform the converged synthetic confidence measure into a structured context and input it into a large language model to generate a visualization script, so as to constrain the visualization elements to be consistent with the confidence of multimodal data.

[0049] The rendering and visualization output module is used to render two-dimensional or three-dimensional visualization layers of buildings based on the generated visualization script, annotate spatial location, status information and confidence level, and form visualization output that can be directly used for building management and monitoring.

[0050] Preferably, the data acquisition and preprocessing module includes:

[0051] The system acquires environmental and image data of the building through sensor and visual acquisition devices, and transmits the data to downstream modules after standardization processing.

[0052] The sensor data is interpolated for missing values, outlier detection is performed based on statistical methods, and corrected sensor data is generated.

[0053] The acquired image data is enhanced and features are extracted. The extracted visual features are then mapped to the sensor data space to generate a visual data confidence measure.

[0054] Preferably, the multimodal joint alignment module includes:

[0055] Joint alignment optimization is performed based on sensor confidence measures and visual confidence measures to ensure the consistency and spatial alignment of multimodal data;

[0056] Based on the characteristics of sensor and visual data, an optimization algorithm is used to determine a mapping function to effectively connect visual data and sensor data.

[0057] Introduce time constraints for the alignment of sensor data and vision data to ensure smooth changes in the data over time.

[0058] The present invention has the following beneficial effects:

[0059] 1. In this invention, multimodal acquisition of building sensor data and visual data is performed, feature fusion is used to generate synthetic confidence measures and then used for visualization rendering. This improves the problem of incomplete status monitoring caused by the inability to integrate multimodal information in traditional building management, which uses a single data source.

[0060] 2. In this invention, by constructing a multimodal joint confidence alignment target and introducing time smoothing constraints, the continuity of sensor data and visual data in the time series is guaranteed, thereby improving the problem that traditional building data processing mostly uses discrete data processing, which lacks time consistency constraints and thus cannot accurately reflect dynamic state changes.

[0061] 3. In this invention, the synthesis confidence measure and mapping function parameters are solved iteratively by alternating minimization strategies, thereby achieving spatial consistency fusion of sensor and visual data. This improves the problem that traditional building visualization generation mostly uses independent data rendering, which cannot achieve spatial alignment and thus causes inaccurate labeling of building spatial information.

[0062] 4. In this invention, by transforming the synthetic confidence measure into a structured context and inputting it into a large language model to generate a visualization script, the visualization elements are constrained to be consistent with the confidence of multimodal data. This improves the problem that traditional building visualization outputs are mostly generated using fixed templates, and the visualization results lack intuitiveness because they cannot combine multimodal confidence information. Attached Figure Description

[0063] Figure 1 This is a flowchart of a method for building data processing and visualization generation proposed in this invention;

[0064] Figure 2 This invention provides a sensor data processing flow for a building data processing and visualization generation method.

[0065] Figure 3 This invention provides a visual data processing flow for a building data processing and visualization generation method.

[0066] Figure 4 This diagram illustrates the iterative solution and optimization process of a building data processing and visualization generation method proposed in this invention.

[0067] Figure 5 This is a flowchart illustrating the visualization generation and rendering process of a building data processing and visualization generation method proposed in this invention.

[0068] Figure 6 This is a module architecture diagram of a building data processing and visualization generation system proposed in this invention. Detailed Implementation

[0069] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] Example 1:

[0071] In a first embodiment of the present invention, the present invention provides a method and system for building data processing and visualization generation, such as... Figures 1-5 As shown, it includes the following steps:

[0072] S1. Collect building sensor data and visual data, perform time series preprocessing and anomaly detection on the sensor data and generate sensor confidence measures, and extract features from the visual data and map them to the sensor reference space to generate visual confidence measures.

[0073] Furthermore, generating sensor confidence measures includes:

[0074] Linear or polynomial interpolation is performed on the collected building sensor time series data to fill in missing data;

[0075] Statistical anomaly detection is applied to the interpolated data, including mean-variance determination and sliding window detection, to remove outlier sampling points;

[0076] The confidence value of each sensor is calculated based on the stability of historical sensor data and the fluctuation of real-time response.

[0077] The processed sensor data and corresponding confidence values ​​are standardized and time-aligned to generate a sensor confidence measurement data matrix in a unified format.

[0078] Generate visual confidence measures include:

[0079] Feature extraction is performed on the collected building visual data using convolutional neural networks or visual language models, including edges, structural textures, and semantic tags;

[0080] Based on the building space topology or CAD model information, visual features are mapped to the sensor reference coordinate system through affine or projection transformations.

[0081] Calculate the confidence score for the mapped visual features to generate a visual confidence measure;

[0082] The visual confidence measure is processed by time series analysis and spatial grid division to form a data matrix that can be directly used for joint alignment.

[0083] Specifically, the sensor confidence metric is generated, and the input data is: a building sensor time series data matrix, where represents the sampled value of the i-th sensor over the time series length.

[0084] Missing value imputation involves linear or polynomial interpolation of missing values ​​in a time series. For linear interpolation, the formula is: ; where represents the interpolated data point, and are the time indices of the sampled points, and is the time of the missing point. Polynomial interpolation can use quadratic or cubic polynomials, in the form: ; where are the fitting coefficients, solved using the least squares method.

[0085] Anomaly detection involves applying statistical anomaly detection methods to the interpolated data. Mean-variance determination is also performed.

[0086] If so, it is judged as abnormal and removed. Here, and represent the historical sampling mean and standard deviation of the sensor, respectively, and represents the upper limit of the standardization deviation.

[0087] Sliding window detection: Data points exceeding a preset threshold are removed. Where is the sliding window length and is the average value of the window.

[0088] Confidence value calculation: The confidence value of each sensor is calculated based on the stability of historical data and the fluctuation of real-time response: where is the confidence level, is the standard deviation of the current time period, and is the historical standard deviation.

[0089] Standardization and time alignment normalize the processed sensor data and corresponding confidence values:

[0090] Output the standardized sensor confidence measure matrix:

[0091] Complete time series alignment to ensure the matrix can be jointly processed with the visual confidence measure matrix in subsequent steps.

[0092] Output data: Standardized sensor confidence measure matrix.

[0093] Visual confidence metric generation, input data: building visual dataset, each frame of image corresponds to the spatial location of the building.

[0094] Feature extraction is performed using convolutional neural networks or visual language models; where is the feature dimension, and is the trained or pre-trained model. Features include edge information, structural texture, and semantic tags.

[0095] Mapping to the sensor reference space, based on the building space topology or CAD model information, using affine or projection transformation: where the corresponding sensor reference coordinate system is.

[0096] Confidence calculation: Calculate the confidence score for the mapped visual features; output the visual confidence measure matrix.

[0097] The time series processing and spatial grid division are performed on the matrix to align it with the time axis of the sensor data; the space is divided into grid cells to form the final visual confidence measure data matrix that can participate in joint alignment.

[0098] Data flow: Input: Raw sensor data matrix and visual dataset. Processing: Sensor data processing to generate; visual data processing to generate. Output: Sensor confidence metric matrix and visual confidence metric matrix, for use by the subsequent joint alignment module.

[0099] S2. Using the generated sensor confidence measure and the mapped visual confidence measure as input, construct a multimodal joint confidence alignment target and determine a learnable mapping function. At the same time, introduce a time smoothing constraint to ensure the continuity of multimodal data in the time series.

[0100] Furthermore, the construction of a multimodal joint confidence alignment objective includes:

[0101] Using the sensor confidence measure matrix and the visual confidence measure matrix as input, a joint confidence alignment objective function is constructed.

[0102] Apply spatial consistency constraints to the objective function to ensure spatial alignment between the sensor and visual data;

[0103] Apply statistical consistency constraints to ensure that the fused data maintains a consistent confidence distribution;

[0104] Apply regularization constraints to the mapping function to ensure spatial continuity;

[0105] Apply a sliding window smoothing constraint to the time series data to ensure time continuity.

[0106] Specifically, after obtaining the sensor confidence metric matrix and the visual confidence metric matrix, the process proceeds to the multimodal joint confidence alignment stage. The main task of this stage is to construct a joint confidence alignment objective function and determine a learnable mapping function to establish a confidence-consistent correlation between the sensor space and the visual space, while maintaining sequence continuity in the time dimension.

[0107] Input data includes a sensor confidence metric matrix; where is the number of sensors and is the number of time frames. Each element in the matrix represents the confidence value of the i-th sensor in the i-th time frame. The visual confidence metric matrix is ​​denoted as , where is the number of visual feature points, and each element in the matrix represents the confidence value of the i-th visual feature point in the i-th frame.

[0108] The output data, the fused confidence matrix after joint confidence alignment, is denoted as , where represents the comprehensive confidence of the i-th spatial location in the i-th time frame after joint alignment.

[0109] To construct the joint confidence alignment objective function and achieve the unification of multimodal information at both the spatial and confidence levels, a joint confidence alignment objective function with respect to the parameters is defined:

[0110] Here, represents the learnable mapping function from visual space to sensor space, and is the set of learnable parameters for this mapping function; represents the Euclidean difference between the sensor confidence measure and the mapped visual confidence measure, used to ensure alignment accuracy; is the spatial consistency constraint term, used to limit the deviation between the mapped visual point and the sensor physical coordinates; is the statistical consistency constraint term, used to constrain the consistent distribution of the fused confidence values; is the regularization constraint term of the mapping function, preventing parameter overfitting and maintaining spatial continuity; is the temporal smoothing constraint term, used to maintain the confidence smoothness of the time series; and is the weight coefficient of each constraint term.

[0111] Spatial consistency constraints are implemented by establishing a mapping relationship between visual features and sensor nodes in the building's three-dimensional coordinate system. Visual feature points and sensor points are defined, with the constraint term: ; where is an affine or projective transformation matrix used to map the visual feature points to the sensor coordinate space. This constraint ensures the alignment accuracy of the two types of data in physical space.

[0112] Statistical consistency constraint: Statistical consistency is used to calibrate the bias in the confidence value distribution of multimodal data. The difference in mean and variance between the two modal confidence distributions is calculated as follows: where is the mean of the sensor confidence matrix; is the mean of the visual confidence matrix; and are the variances of the two, respectively. This constraint ensures statistical consistency in the confidence distributions of the two types of data.

[0113] Mapping function regularization constraint: To prevent spatial jumps in the mapping function, a parametric regularization form is adopted: ; where represents the gradient of the mapping function with respect to the input visual features, used to constrain the continuity and stability of the mapping.

[0114] Time smoothing constraint introduces a sliding window smoothing term into time series data:

[0115] Here, represents the fused confidence matrix for the i-th frame. This constraint reduces abrupt changes between time frames, ensuring that the confidence level changes smoothly over time.

[0116] The mapping function is determined and output by minimizing the joint objective function constructed above: The optimal parameters of the mapping function are obtained. After optimization, the fused confidence matrix is ​​obtained, where represents the mapping function parameters determined after convergence of iterative optimization. This fusion result is fed into the subsequent iterative solution stage as the joint input of multimodal confidence, providing an alignment basis for subsequent synthetic confidence updates and visualization script generation.

[0117] S3. Iteratively solve the synthetic confidence measure and mapping function parameters by alternating minimization strategies until the joint optimization objective converges, and update the synthetic confidence measure in each iteration to fuse visual and sensor information;

[0118] Furthermore, the parameters of the synthetic confidence measure and mapping function include:

[0119] The synthesized confidence measure is initialized as a weighted average of the sensor confidence measures, and the mapping function parameters are initialized as unit mapping or random perturbation;

[0120] With fixed mapping function parameters, the synthetic confidence measure is updated by minimizing the joint confidence alignment objective.

[0121] With a fixed synthetic confidence measure, update the mapping function parameters using gradient descent or alternating minimization strategies;

[0122] Repeat the steps until the joint objective function converges or the upper limit of iteration is reached;

[0123] Output the converged synthetic confidence measure matrix to provide a data foundation for visualization generation.

[0124] Specifically, the system uses an aligned sensor confidence metric matrix. and visual confidence measure matrix As input data, the synthetic confidence measure is evaluated using an alternating minimization strategy. The combined iterative solution of the mapping function parameter set yields the fused synthetic confidence measure output matrix, which is used in the subsequent visualization script generation stage. This represents the spatial node dimension of the synthetic confidence measure, and its value is related to the sensor confidence measure matrix. number of rows To maintain consistency and meet the dimension matching requirements of matrix algebra operations. This represents the number of frames in the time series.

[0125] During the initialization phase, the system first uses the sensor confidence metric matrix. The global statistics are initialized, and the initial composite confidence measure matrix is ​​defined as follows: ;in The confidence weighting coefficient is... This is the column mean matrix of the sensor confidence measure. The set of parameters for the mapping function. Initialize it to a unit mapping parameter or a random mapping matrix with a small perturbation to ensure the feasibility of subsequent gradient calculations.

[0126] Solving for the composite confidence measure with fixed mapping function parameters, at the th In each iteration, the set of parameters of the mapping function is maintained. The objective function remains unchanged; it is aligned by minimizing the joint confidence. The synthetic confidence measure is updated. The objective function is defined as: ;in This is a data fitting constraint term used to constrain the synthetic confidence measure. The formula for approximating the weighted fusion value of multimodal data under the current parameters is as follows:

[0127] ;

[0128] in This represents the square of the Euclidean distance. For the sensor confidence measure matrix, For parameter-based The mapped visual confidence measure matrix.

[0129] This is a statistical consistency constraint term used to constrain the composite confidence level. The distribution characteristics. Its calculation logic references the mean-variance determination method described earlier, specifically:

[0130] ;

[0131] in , The confidence measure matrix is ​​synthesized in the current iteration step. The global mean and standard deviation; , As a reference statistic, the value is taken as the average of the statistical means of the sensor data and the visual data. This constraint ensures the quality of the synthetic data. There will be no drift in the statistical distribution, maintaining consistency with multimodal source data.

[0132] This is a time smoothing constraint term used to ensure the continuity of confidence levels over time. The specific formula is as follows: ;in Representation matrix In the The confidence vectors of all nodes in each time frame. The synthetic confidence measure is updated using a gradient descent-based update method: .

[0133] The parameters of the composite confidence measure update mapping function are kept constant. Unchanged, through the Minimize and update the parameter set of the mapping function:

[0134] ;

[0135] in Update the step size for the mapping parameters.

[0136] Mapping function parameters It includes weight parameters that define the spatial mapping relationship. Its gradient update direction is determined by the parameter-related constraints in the joint objective function. The first term is the gradient of the data fitting constraint with respect to the parameters, which drives the mapping function to achieve spatial alignment between visual and sensor data; the second term is the gradient of the regularization constraint with respect to the parameters, which smooths the parameter distribution and prevents the mapping function from overfitting.

[0137] For iterative convergence determination, the system calculates the relative rate of change of the objective function after each iteration:

[0138] ;

[0139] when or number of iterations Reaching the preset limit When the iteration stops, the iteration continues. This is the convergence threshold, used to determine whether the joint optimization is stable.

[0140] In the output phase, the converged composite confidence measure matrix is ​​output. This matrix serves as input data for the subsequent visualization generation module. The system output... The confidence continuity in the time and spatial dimensions is preserved after multimodal fusion. The output format is structured matrix data, with fields including: time index, spatial location index, fusion confidence value, and mapping parameter reference index.

[0141] Input and output flow, input: sensor confidence metric matrix Visual confidence measure matrix Mapping initial parameters Processing flow: Initialize the synthesized confidence measure; update the confidence matrix while fixing the mapping parameters; update the mapping parameters while fixing the confidence matrix; determine the convergence condition; output the convergent synthesized confidence matrix. Output: Synthetic confidence measure matrix. and the corresponding set of mapping function parameters .

[0142] S4. The converged synthetic confidence measure is transformed into a structured context and input into a large language model to generate a visualization script to constrain the visualization elements to be consistent with the confidence of the multimodal data.

[0143] Furthermore, generating the visualization script includes:

[0144] The converged synthetic confidence measure matrix is ​​transformed into a structured context, including node locations, state attributes, and confidence information.

[0145] Input structured context into a large language model, and generate visualization scripts through templates or instructions to make the spatial position and confidence of visualization elements correspond;

[0146] The generated script is parsed to map two-dimensional or three-dimensional graphic elements to the building coordinate system.

[0147] Specifically, in step S3, the synthesized confidence measure matrix is ​​obtained. This represents the confidence level of multimodal information fusion across the spatiotemporal dimensions. Each element... Corresponding to the The sensor observation and the first The fusion confidence values ​​between visual nodes. To achieve data integration with the visualization output system, this matrix needs to be structured and encoded to form a context input that the language model can parse. The process of constructing the structured context includes: ;in This represents the three-dimensional spatial position of the node in the building coordinate system; For the node's state attributes, such as device status or signal type; Depend on The confidence value extracted corresponds to the confidence measure of that node.

[0148] This structured data It is fed into a large language model as input. The model is based on a predefined visualization script template. Generate semantically consistent script output: ;in Indicates the output of the visualization script; template The rules for graphical elements, coordinate binding, and confidence mapping are defined.

[0149] The model output script uses a structured syntax (such as JSON or a custom markup language) and includes fields such as node coordinates, shape type, color depth, and transparency. The color or transparency parameters have a monotonic mapping relationship with the confidence value, for example: ;in Visual transparency of nodes; These represent the minimum and maximum values ​​in the confidence matrix. This mapping ensures that high-confidence nodes are highlighted in the visualization, while low-confidence nodes are displayed semi-transparently.

[0150] The parsing module maps the generated two-dimensional or three-dimensional graphic elements to the building coordinate system based on the script content. This achieves spatial alignment and structural constraints. The mapping process is defined as follows:

[0151] ;

[0152] in Represents the set of visualized elements after projection; This represents the rendering and coordinate transformation function, which performs different mapping processes depending on the output target (2D or 3D): for 3D visualization layers : Perform model transformation through affine transformation matrix Map the local coordinates of elements to the building's 3D world coordinate system. ,Right now .

[0153] For 2D visualization layers : Based on the above model transformation, a further view projection transformation is performed using the view matrix. and projection matrix Mapping three-dimensional world coordinates to a two-dimensional view plane coordinate system, i.e. Through the above-described hierarchical mapping process, the system achieves a closed-loop data flow from the confidence fusion result to visualizations across different dimensions: .

[0154] Through the above process, the system achieves a closed-loop data flow from confidence fusion results to visualization:

[0155] .

[0156] By transforming the converged synthetic confidence measure matrix into a structured context, a mapping mechanism from numerical confidence output to semantic visualization input is established, enabling multimodal data to have parsability and consistency.

[0157] By introducing a templated visual script generation method into large-scale language models, we can ensure the consistency of expression of different confidence nodes at the script level, thereby reducing the complexity of manual annotation and rule writing.

[0158] By leveraging the monotonic mapping relationship between confidence level and visual attributes, a quantitative correspondence between confidence information and visualization is achieved, enabling the generated visualization results to accurately reflect the credible distribution after sensor and vision fusion.

[0159] By parsing the script and performing coordinate mapping, the spatial consistency of the generated graphics in the building coordinate system is ensured, providing an input basis for subsequent projection display or digital twin environment rendering.

[0160] S5. Render two-dimensional or three-dimensional visualization layers of the building based on the generated visualization script, mark the spatial location, status information and confidence level, and form a visualization output that can be directly used for building management and monitoring.

[0161] Furthermore, rendering two-dimensional or three-dimensional visualization layers of buildings includes:

[0162] Receive element information generated by a structured context or a visualization script;

[0163] Map node information to a two-dimensional or three-dimensional coordinate system of the building;

[0164] Render visual elements, including node position, shape, and color encoding;

[0165] Generate legends or labels based on node attributes and confidence levels;

[0166] Output complete visualization layers or datasets for building management or monitoring.

[0167] Specifically, after completing step S4, the system has obtained a structured visual script. This includes node coordinates, confidence values, status attributes, and display rules. Step S5 uses this script to render layers and generate two-dimensional or three-dimensional visualizations of the building.

[0168] Data input includes: ;in This represents the spatial location of the node in the building coordinate system; This refers to the node's state attributes. The node confidence value; For color or transparency parameters; The element type is specified (point, line, surface, or volume). The system maps each node to the building's geometric model coordinate system based on the coordinate data. The mapping function is:

[0169] ;

[0170] in This is the coordinate transformation matrix from the sensing space to the building space, including the rotation matrix. With translation vector ,Right now: ;in .

[0171] During the rendering phase, the system generates visual codes based on node attributes and confidence values. The color coding function is defined as: ;in The RGB color of the node; This is the mapping function from confidence level to color space; These are the color component values ​​corresponding to the confidence level interval. Color encoding uses a linear or piecewise mapping rule, so that nodes with high confidence are displayed in bright colors, and nodes with low confidence are displayed in light colors. Node size. or transparency Calculated separately using the following independent functions:

[0172] ;

[0173] ;

[0174] in For node dimensions; To determine node transparency; This is a mapping function between confidence level and displayed attributes.

[0175] Based on node type The system calls the point rendering, line rendering, or surface rendering modules respectively to generate the corresponding geometric entities. The specific rendering process involves parsing the visualization script to obtain the element set. The set is defined as follows: ;in This refers to the geometric vertex data of the element in the world coordinate system. For visual attributes, This corresponds to the fusion confidence value. The rendering module determines this based on... Composite with the building model to generate output layers. This is for 2D floor plans. Layer generation: Geometric projection: using the view projection moments determined in the preceding steps ,Will The three-dimensional vertices of all elements and building models are projected onto a two-dimensional view plane to obtain the screen pixel coordinates.

[0176] Rasterization: Discretizing the projected geometric primitives into pixel fragments.

[0177] Compositing and blending: Depth buffering technology is used to handle occlusion relationships, ensuring the correct spatial relationship between the building structure and the monitored elements; Alpha blending technology is used to overlay the monitored element layer onto the building base map, with the blending formula being... ,in From confidence level Decision. The final output is a two-dimensional visual planar layer containing spatial annotation information. .

[0178] After the layer is generated, the system determines its state based on the node attributes. With confidence value Automatically generate a set of illustrations: ;in For status labels; This corresponds to the confidence interval or state range.

[0179] The final output includes: This output can be directly used as input for the building management system interface for real-time status display, monitoring alarms, or interactive operations.

[0180] By receiving and parsing structured data in visualization scripts, the system establishes a mapping process from semantic scripts to physical space layers, enabling accurate visualization of confidence data in the building coordinate system.

[0181] By mapping confidence levels to color, transparency, and size, a quantifiable visual hierarchical structure is formed, enabling the monitoring interface to intuitively reflect the credibility status and anomaly level of each node.

[0182] By defining a coordinate mapping matrix This achieves geometric alignment between the multimodal acquisition space and the building space, ensuring that the rendering results correspond to the real spatial structure.

[0183] By automatically generating legends and annotations, a complete data interpretation system is formed, providing a directly usable visualization interface for the building management system, making it easier for operation and maintenance personnel to identify and track multimodal monitoring results.

[0184] The rendered layer data has a structured interface that can be called by upper-level monitoring platforms or digital twin engines to achieve dynamic overlay and interactive display.

[0185] Example 2:

[0186] In large commercial complexes or smart office buildings, management needs to monitor and visualize the building's internal environment, equipment operating status, and energy consumption in real time. The system collects data such as temperature, humidity, air quality, lighting, and power consumption through building sensors, and simultaneously acquires spatial visual information, including pedestrian distribution, equipment appearance, and building structure, using cameras or laser scanning. Existing systems face challenges in multimodal data fusion in this scenario: sensor and visual data are inconsistent in time and space, leading to data alignment difficulties; different modal data have significantly different confidence levels, lacking a unified quantification method; real-time fusion in dynamic environments lacks effective temporal continuity constraints and iterative optimization mechanisms; during the conversion of multimodal fusion results into visualization scripts, consistency in spatial mapping, node attributes, and confidence levels is difficult to guarantee; when rendering 2D or 3D layers, node positions, shapes, colors, and annotations must strictly correspond to the fused data, otherwise misinterpretation of information may occur. To address these issues, this invention provides a building data processing and visualization generation system, the structure of which is as follows: Figure 6 As shown. The specific implementation process of this system is as follows:

[0187] Specifically, during the process of rendering two-dimensional or three-dimensional visualization layers of buildings based on the generated visualization script, by receiving the structured context or element information generated by the visualization script, the system can ensure that spatial mapping and layer construction are performed based on the structured description generated in the previous steps during the rendering stage, thus ensuring the consistency and traceability of building elements in the visualization space.

[0188] Mapping node information to a two-dimensional or three-dimensional coordinate system of a building enables accurate positioning of various functional units within the building space, providing a spatial reference standard for subsequent visualization outputs and offering a unified spatial benchmark for building structure, equipment layout, and status labeling.

[0189] When rendering visual elements, by processing the node position, shape and color encoding, different types or states of devices and areas are presented in a differentiated way in the layer, so that operators can distinguish multiple types of operational information within the same interface and realize an intuitive display of building management information.

[0190] Legends or labels are generated based on node attributes and confidence levels, which can reflect the reliability and status level of the data source on the visualization layer, ensuring that monitoring and management personnel can identify the credibility level of each node and make maintenance or scheduling decisions accordingly.

[0191] The final output is a complete visualization layer or dataset, enabling the system to directly use the visualization results for building management or monitoring, realizing a closed loop from data structuring to spatial visualization, supporting real-time display of building operation status and intuitive prompts for abnormal information, and improving the operability and usability of the building information system.

[0192] The data acquisition and preprocessing module includes:

[0193] The system acquires environmental and image data of the building through sensor and visual acquisition devices, and transmits the data to downstream modules after standardization processing.

[0194] The sensor data is interpolated for missing values, outlier detection is performed based on statistical methods, and corrected sensor data is generated.

[0195] The acquired image data is enhanced and features are extracted. The extracted visual features are then mapped to the sensor data space to generate a visual data confidence measure.

[0196] Specifically, the data acquisition and preprocessing module obtains environmental data about the building through sensor devices, including temperature, humidity, air quality, and energy consumption, and acquires images of the building space through visual acquisition devices. The acquired data is standardized and transmitted to downstream modules in a unified format to ensure consistent input in time and space for subsequent processing.

[0197] During sensor data processing, missing values ​​are interpolated to fill data gaps, and statistical methods are used to detect and remove outliers, thereby generating a corrected sensor data matrix. This step ensures that the input data is complete and consistent enough for multimodal fusion, while also providing confidence information for each sensor data point for downstream calculations.

[0198] In visual data processing, image enhancement is performed on the acquired images to improve feature extractability, and visual feature vectors are generated using convolutional neural networks or visual feature extraction algorithms. Subsequently, the extracted visual features are mapped to the sensor reference space based on building spatial topology or sensor coordinates, forming a visual data confidence metric matrix. This matrix provides a spatial consistency basis for multimodal alignment, enabling joint processing of visual information and sensor data under unified spatial coordinates.

[0199] The entire module realizes the complete process from raw multi-source data acquisition to standardization, correction and spatial mapping, providing directly usable input data for subsequent multimodal joint alignment and visualization generation.

[0200] The multimodal joint alignment module includes:

[0201] Joint alignment optimization is performed based on sensor confidence measures and visual confidence measures to ensure the consistency and spatial alignment of multimodal data;

[0202] Based on the characteristics of sensor and visual data, an optimization algorithm is used to determine a mapping function to effectively connect visual data and sensor data.

[0203] Introduce time constraints for the alignment of sensor data and vision data to ensure smooth changes in the data over time.

[0204] Specifically, the multimodal joint alignment module establishes a joint alignment objective for multimodal data based on sensor confidence measures and visual confidence measures. It processes the input data through optimization algorithms to achieve spatial and feature consistency across different modalities. By defining a joint alignment objective function, the module incorporates the spatial relationship and confidence information between sensor data and visual features into the computational framework, providing a standardized foundation for subsequent fusion.

[0205] During the determination of the mapping function, the module calculates the mapping parameters based on the feature vectors of the sensor and visual data through iterative optimization or minimization strategies, accurately mapping the visual data to the sensor reference coordinate system. The mapping function aligns visual information and sensor data spatially and generates a joint confidence measure matrix that can be directly used for multimodal fusion, providing the input data foundation for the iterative solution module.

[0206] The module introduces time constraints into the joint alignment objective, smoothing the changes in sensor and visual data over time to ensure data continuity. Through sliding windows or time smoothing constraints, the data maintains logical consistency at different time points, supporting continuous monitoring and visualization output in dynamic environments.

[0207] The overall module realizes the complete process of multimodal data from feature extraction to spatial alignment and temporal consistency processing, providing directly usable multimodal inputs for subsequent iterative solutions to fused confidence measures and visualization generation.

[0208] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for building data processing and visualization generation, characterized in that, Includes the following steps: S1. Collect building sensor data and visual data, perform time series preprocessing and anomaly detection on the sensor data and generate sensor confidence measures, and extract features from the visual data and map them to the sensor reference space to generate visual confidence measures. S2. Using the generated sensor confidence measure and the mapped visual confidence measure as input, construct a multimodal joint confidence alignment target and determine a learnable mapping function. At the same time, introduce a time smoothing constraint to ensure the continuity of multimodal data in the time series. S3. Iteratively solve the synthetic confidence measure and mapping function parameters by alternating minimization strategies until the joint optimization objective converges, and update the synthetic confidence measure in each iteration to fuse visual and sensor information; S4. The converged synthetic confidence measure is transformed into a structured context and input into a large language model to generate a visualization script to constrain the visualization elements to be consistent with the confidence of the multimodal data. S5. Render two-dimensional or three-dimensional visualization layers of the building based on the generated visualization script, mark the spatial location, status information and confidence level, and form a visualization output that can be directly used for building management and monitoring.

2. The method for building data processing and visualization generation according to claim 1, characterized in that, The generated sensor confidence measure includes: Linear or polynomial interpolation is performed on the collected building sensor time series data to fill in missing data; Statistical anomaly detection is applied to the interpolated data, including mean-variance determination and sliding window detection, to remove outlier sampling points; The confidence value of each sensor is calculated based on the stability of historical sensor data and the fluctuation of real-time response. The processed sensor data and corresponding confidence values ​​are standardized and time-aligned to generate a sensor confidence measurement data matrix in a unified format.

3. The method for building data processing and visualization generation according to claim 1, characterized in that, The generated visual confidence measure includes: Feature extraction is performed on the collected building visual data using convolutional neural networks or visual language models, including edges, structural textures, and semantic tags; Based on the building space topology or CAD model information, visual features are mapped to the sensor reference coordinate system through affine or projection transformations. Calculate the confidence score for the mapped visual features to generate a visual confidence measure; The visual confidence measure is processed by time series analysis and spatial grid division to form a data matrix that can be directly used for joint alignment.

4. The method for building data processing and visualization generation according to claim 1, characterized in that, The objectives for constructing the multimodal joint confidence alignment include: Using the sensor confidence measure matrix and the visual confidence measure matrix as input, a joint confidence alignment objective function is constructed. Apply spatial consistency constraints to the objective function to ensure spatial alignment between the sensor and visual data; Apply statistical consistency constraints to ensure that the fused data maintains a consistent confidence distribution; Apply regularization constraints to the mapping function to ensure spatial continuity; Apply a sliding window smoothing constraint to the time series data to ensure time continuity.

5. The method for building data processing and visualization generation according to claim 1, characterized in that, The parameters of the synthetic confidence measure and mapping function include: The synthesized confidence measure is initialized as a weighted average of the sensor confidence measures, and the mapping function parameters are initialized as unit mapping or random perturbation; With fixed mapping function parameters, the synthetic confidence measure is updated by minimizing the joint confidence alignment objective. With a fixed synthetic confidence measure, update the mapping function parameters using gradient descent or alternating minimization strategies; Repeat the steps until the joint objective function converges or the upper limit of iteration is reached; Output the converged synthetic confidence measure matrix to provide a data foundation for visualization generation.

6. The method for building data processing and visualization generation according to claim 1, characterized in that, The generated visualization script includes: The converged synthetic confidence measure matrix is ​​transformed into a structured context, including node locations, state attributes, and confidence information. Input structured context into a large language model, and generate visualization scripts through templates or instructions to make the spatial position and confidence of visualization elements correspond; The generated script is parsed to map two-dimensional or three-dimensional graphic elements to the building coordinate system.

7. The method for building data processing and visualization generation according to claim 1, characterized in that, The rendered 2D or 3D visualization layer of the building includes: Receive element information generated by a structured context or a visualization script; Map node information to a two-dimensional or three-dimensional coordinate system of the building; Render visual elements, including node position, shape, and color encoding; Generate legends or labels based on node attributes and confidence levels; Output complete visualization layers or datasets for building management or monitoring.

8. A system for building data processing and visualization generation, characterized in that, The method for building data processing and visualization generation according to any one of claims 1-7 includes the following modules: The data acquisition and preprocessing module is used to acquire building sensor data and visual data, perform time series preprocessing and anomaly detection on sensor data and generate sensor confidence measures, and extract features from visual data and map them to sensor reference space to generate visual confidence measures. The multimodal joint alignment module is used to construct a multimodal joint confidence alignment target by taking the generated sensor confidence measure and the mapped visual confidence measure as input, and to determine the learnable mapping function. At the same time, a time smoothing constraint is introduced to ensure the continuity of multimodal data in the time series. The iterative solution module is used to iteratively solve the synthetic confidence measure and mapping function parameters by alternating minimization strategies until the joint optimization objective converges, and updates the synthetic confidence measure in each iteration to fuse visual and sensor information; The visualization script generation module is used to transform the converged synthetic confidence measure into a structured context and input it into a large language model to generate a visualization script, so as to constrain the visualization elements to be consistent with the confidence of multimodal data. The rendering and visualization output module is used to render two-dimensional or three-dimensional visualization layers of buildings based on the generated visualization script, annotate spatial location, status information and confidence level, and form visualization output that can be directly used for building management and monitoring.

9. The system for building data processing and visualization generation according to claim 8, characterized in that, The data acquisition and preprocessing module includes: The system acquires environmental and image data of the building through sensor and visual acquisition devices, and transmits the data to downstream modules after standardization processing. The sensor data is interpolated for missing values, outlier detection is performed based on statistical methods, and corrected sensor data is generated. The acquired image data is enhanced and features are extracted. The extracted visual features are then mapped to the sensor data space to generate a visual data confidence measure.

10. The system for building data processing and visualization generation according to claim 8, characterized in that, The multimodal joint alignment module includes: Joint alignment optimization is performed based on sensor confidence measures and visual confidence measures to ensure the consistency and spatial alignment of multimodal data; Based on the characteristics of sensor and visual data, an optimization algorithm is used to determine a mapping function to effectively connect visual data and sensor data. Introduce time constraints for the alignment of sensor data and vision data to ensure smooth changes in the data over time.

Images