Full-scene real-time energy consumption simulation optimization method and system based on AR (Augmented Reality) equipment

Abstract

The invention provides a full-scene real-time energy consumption simulation optimization method and system based on AR (Augmented Reality) equipment. The method comprises the following steps: collecting visual data and depth-of-field data of a physical scene; fusing the preprocessed visual data and depth-of-field data to construct a three-dimensional point cloud model; performing energy consumption category segmentation and equipment instance segmentation on the collected visual data, and constructing a segmentation optimization mechanism to improve the accuracy of a segmentation result; constructing an energy consumption equipment database to realize accurate identification of energy consumption equipment in the scene; and realizing visualization based on an AR interface. According to the method, a double-model parallel architecture is adopted, pixel-level category labeling is carried out on an input image, category masks are output, the image is segmented into areas with different category meanings, each pixel is endowed with a specific category label, the equipment recognition accuracy is improved, strong binding of the equipment category labels and point cloud positions during subsequent space alignment is ensured, and the accuracy of equipment recognition is improved. And reliable equipment information is provided for energy consumption analysis.

Description

Technical Field

[0001] This invention relates to the field of energy consumption simulation technology, specifically to a method and system for real-time energy consumption simulation and optimization across all scenarios based on AR devices. Background Technology

[0002] With the increasing demands for energy efficiency in the industrial and building sectors, energy consumption monitoring and optimization systems are playing an increasingly important role in practical applications.

[0003] Currently, regarding database construction, patent CN118247781A proposes a deep learning-based industrial robot target recognition method and system. It aims to construct an industrial robot recognition model using 3D point cloud data acquired by deep learning technology and depth cameras, solving the problem of target recognition in complex scenarios. However, it has several significant shortcomings in the field of industrial robot target recognition. ① In terms of data acquisition, relying solely on depth cameras to obtain point cloud data easily leads to misjudgments in practical applications. When facing motors / other equipment of different models but similar shapes, it is difficult to distinguish them based solely on point cloud data, significantly increasing the misjudgment rate and affecting data accuracy. ② In the model building stage, its cascaded model architecture exposes the problem of losing detailed features during data transmission. Data transmission from the semantic model to the target discrimination model requires downsampling (Ns×3→Ns×32), resulting in the loss of detailed features, such as object texture features, affecting the accuracy of the discrimination model and consequently impacting the real-time performance and accuracy of the entire system. ③ In the scenario database construction stage, the semantic granularity is insufficient, and the distinction between equipment and non-equipment entities is not effectively made. This results in a lack of accurate identification and classification of specific equipment entities in the data, leading to errors due to inaccurate or untimely data updates. When the physical attributes of the equipment change (such as equipment upgrades or replacement of parts), the database cannot reflect these changes in a timely manner, causing the analysis results based on the database to be inconsistent with the actual situation, affecting the accuracy and reliability of the data.

[0004] In the area of ​​energy consumption management, CN119167792A proposes an adaptive calculation method, system, equipment, and medium for building heating and cooling loads. Through computer vision and machine learning technologies, combined with building information modeling, it solves the data acquisition and accuracy problems of traditional methods, achieving high-precision and adaptive calculation of building heating and cooling loads, thus improving the efficiency of energy management and supporting sustainable building development. However, it has the following logical flaws in the field of building heating and cooling load prediction: ① Inadequate post-segmentation processing: After extracting image features using edge detection, feature point recognition, and semantic segmentation techniques, the segmented data was not further validated or optimized for subsequent model improvements. For example, using the Mask R-CNN model to semantically segment the image data of air conditioning equipment inside a building, the location and shape of the equipment could be identified. However, without further validation and optimization of the segmentation results, the segmented data suffered from blurred boundaries, misidentification, and size deviations. This resulted in the segmented air conditioning equipment area including pixels of non-air conditioning equipment or omitting pixels of some air conditioning equipment, affecting the accuracy of the overall building's heating and cooling load prediction. ② The energy consumption prediction model primarily relied on random forest and neural network algorithms, combined with environmental parameters such as electricity consumption data, equipment power, and solar radiation. However, this approach did not adequately consider how to effectively integrate multimodal data or how to quantify the uncertainties in the prediction process within the model. ③ Finally, in the data fusion and preprocessing stage, the building's image data and electricity consumption data were preprocessed to extract time-series features. However, when performing 3D reconstruction and BIM model generation, these time-series features were not considered to enhance the model's ability to capture dynamic changes in building energy consumption.

[0005] Therefore, there is an urgent need for a real-time energy consumption simulation and optimization system that can achieve unified perception and fusion of spatial location, equipment status and environmental data across all scenarios, in order to support more accurate and comprehensive energy consumption analysis and optimization. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for real-time energy consumption simulation and optimization across all scenarios based on AR devices, thereby resolving the problems mentioned in the background section.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a real-time energy consumption simulation and optimization method for all scenarios based on AR devices, comprising the following steps: By utilizing the multimodal sensor array of AR devices, visual data, depth data, and operational data representing target energy-consuming devices in physical scenes are collected to construct a three-dimensional point cloud model representing the scene where the energy-consuming devices are located. The collected visual data is segmented by energy consumption category and device instance. A segmentation optimization mechanism is constructed to improve the accuracy of the segmentation results and identify energy-consuming device elements in the physical scene. The execution process of the segmentation optimization mechanism is as follows: The overlap ratio between the instance mask output by device instance segmentation and the corresponding category binary mask output by energy consumption category segmentation is used to measure the degree of coverage overlap between the region where the device instance segmentation is located and the region where the corresponding category mask is located. A consistency threshold is set. If the degree of coverage overlap is higher than the threshold, the instance information is retained and output as the structured segmentation result. The process of building an energy-consuming device database to accurately identify energy-consuming devices in a scenario is as follows: After spatial alignment of the optimized segmentation results, 3D point cloud model data, and operational data, they are converted into world coordinate system points to map category and instance information to the 3D point cloud, resulting in a 3D point cloud set of instance information. Next, the centroid of the 3D point cloud is calculated using a weighted average and used as the spatial coordinates of the target device. Finally, based on the obtained point cloud and centroid, energy consumption device database entries are created: the 3D point cloud set is traversed, and the minimum and maximum coordinate values ​​of each point cloud in the x, y, and z dimensions are found to form a spatial bounding box, defining the location and range of instance information in space. The category label, unique ID, centroid coordinates, and spatial bounding box parameters of the instance information are stored in the node, constructing the device database entries of the energy consumption simulation system, achieving strong binding between the segmentation results, 3D point cloud model data, and device operational data. Based on the identified energy-consuming equipment information, a spatiotemporal sequence model is constructed to predict energy consumption, and the energy consumption prediction results are visualized using an AR interface.

[0008] As a second aspect of the present invention, a real-time energy consumption simulation and optimization system for all scenarios based on AR devices is proposed, including a memory and a processor. The memory includes a real-time energy consumption simulation and optimization program for all scenarios based on AR devices. When the real-time energy consumption simulation and optimization program for all scenarios based on AR devices is executed by the processor, it implements the above-mentioned real-time energy consumption simulation and optimization method for all scenarios based on AR devices.

[0009] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. To address the issue of misclassification / device instance segmentation caused by similar equipment appearances or occlusion in existing industrial scenarios, this invention, after initial segmentation, constructs hierarchical associations and calculates consistency scores to accurately assess the reliability of the segmentation results. Combined with a secondary verification mechanism based on temporal information, it ensures the stability of energy consumption fluctuations and equipment operating states, thereby filtering potential errors. Then, through continuous incremental learning and parameter optimization, the model can dynamically adjust, significantly reducing the segmentation error rate. The final structured output not only includes high-confidence instance masks and category labels but also incorporates equipment operating parameters, enhancing the model's contextual relevance and stability in energy consumption scenarios. This provides high-precision, strongly correlated visual feature primitives for subsequent spatial alignment and energy consumption analysis, significantly improving the accuracy of equipment identification, reducing the need for manual verification, optimizing model performance, and ensuring the accuracy and reliability of energy consumption prediction.

[0010] 2. This invention uses the NTP protocol to precisely synchronize the clocks of all sensor devices, ensuring that all collected data have a unified time base. Simultaneously, it utilizes bilinear interpolation technology to map discrete data points from different sensors into a unified three-dimensional grid coordinate system, effectively integrating data from different sensors and ensuring spatial consistency. Secondly, in terms of anomaly suppression, a statistical threshold-based method is used to detect and process anomalies, reducing the impact of noise on the entire system and improving data quality and reliability. Finally, an extended Kalman filter algorithm is used for dynamic fusion of multi-source data. Utilizing a prediction and update loop mechanism, real-time estimation and updating of device status and environmental data are achieved, effectively handling uncertainties and noise in the data and further improving the accuracy of data fusion. Attached Figure Description

[0011] The disclosure of this invention is illustrated with reference to the accompanying drawings. It should be understood that the drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. In the drawings, the same reference numerals are used to refer to the same parts. Wherein: Figure 1 This is a schematic diagram of the energy consumption simulation and optimization process proposed in one embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the process of constructing an energy consumption device database including spatial location, visual entities, device IDs, and real-time data streams, as proposed in one embodiment of the present invention. Figure 3 This is a schematic diagram illustrating the process of constructing an incremental learning mechanism to improve the segmentation effect of a segmentation model, as proposed in one embodiment of the present invention. Detailed Implementation

[0012] It is readily understood that, based on the technical solution of this invention, those skilled in the art can propose various interchangeable structural methods and implementations without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention.

[0013] The present invention will be further described in detail below with reference to the accompanying drawings, but this is not intended to limit the scope of the invention.

[0014] like Figure 1 As shown, as an embodiment of the present invention, the first proposed method for real-time energy consumption simulation and optimization based on AR devices in all scenarios includes the following steps: S1. Data Acquisition: Utilizing the multimodal sensor array of the AR device, including a high-resolution RGB camera and a depth sensor, visual data and depth data of the physical scene are acquired simultaneously. The acquired data is then standardized and preprocessed to ensure the accuracy and consistency of the data. S2. Data Fusion: Fusion of preprocessed visual data and depth data to construct a 3D point cloud model representing the physical scene; S3, Energy Consumption Device Database: Using a deep learning model, energy consumption category segmentation and device instance segmentation are performed on RGB images captured by high-resolution RGB cameras in AR devices. Energy consumption device elements in the physical scene are identified, and the segmentation results, 3D point cloud model data and device operation data are spatially aligned to build an energy consumption device database including spatial location, visual entity, device ID, and real-time data stream. This enables accurate identification of energy consumption devices in the scene, thereby providing detailed device information for energy consumption monitoring. S4. Energy Consumption Prediction Simulation: Based on the identified energy-consuming equipment information, a spatiotemporal sequence model is constructed to predict energy consumption, and the Monte Carlo dropout method is used to generate confidence intervals. The uncertainty in the energy consumption prediction process is quantified by combining point prediction values ​​with probability intervals. S5. Continuously track the energy consumption data of various devices in the physical scene. Through the built low-latency data pipeline, continuously receive the operating parameter data of energy-consuming devices pushed by the multi-threaded IoT protocol interface, and perform real-time cleaning and preprocessing, and link it to the energy-consuming device database in real time. S6. Based on the energy consumption prediction results, construct an optimization problem model with the objective function of minimizing energy consumption cost, total energy consumption, or peak demand, output the energy consumption equipment control strategy, and realize visualization based on the AR interface.

[0015] In one embodiment of the present invention, to provide precise positioning for optimizing the energy consumption of industrial equipment in a physical scene, data acquisition is achieved in step S1 through the following steps: S1-1. Utilize the multimodal sensor array of the AR device to synchronously collect data within the physical scene, including: Visual data: Visual data in the physical scene is captured using an industrial-grade RGB camera, and image frames I are acquired at multiple consecutive time points t=1,2,...,M. t To form a time-series image sequence .

[0016] Depth data: Spatial depth data characterizing the relative positions and distances between energy-consuming devices in a physical scene is acquired using ToF / structured light depth sensors. It should be noted that due to the optical characteristics of existing cameras, the captured images will exhibit varying degrees of distortion, mainly radial and tangential distortion, which in turn affects the overall energy consumption optimization effect.

[0017] Therefore, the visual data and depth data obtained in S1 need to be preprocessed to ensure the reliability of subsequent energy consumption analysis and optimization.

[0018] S1-2. Standardize and preprocess the collected data. In practice, the data preprocessing process follows a hierarchical calibration mechanism: First, distortion correction is performed on the visual data: Based on the Zhang calibration model, at least 10 checkerboard pattern images (RGB images) from different angles are captured, and the corner coordinates are detected using the cornerDetect function of OpenCV. The least squares method is used to calculate the intrinsic parameter matrix K of the RGB camera, and radial-tangential distortion correction is performed on the images to reduce image edges and improve the accuracy of shape and position measurement of energy-consuming devices.

[0019] Secondly, spatial registration is performed by establishing a transformation relationship between the visual and depth coordinate systems to achieve precise registration between the depth data and the visual data, ensuring a high degree of consistency between them. Specific steps include: a. Time-series image sequences acquired through synchronous acquisition Spatial depth data Using the focal length in the intrinsic parameter matrix K of the RGB camera and and the principal point coordinates of the image and The pixel coordinates representing the image location Depth values ​​representing the distance information from a pixel to the RGB camera Back projection into 3D point cloud: At the same time, invalid depth points (such as...) are removed. ), generate a set of matching point pairs This facilitates subsequent conversion and registration between the visual and depth coordinate systems using these matching point pairs. For visual feature points, For the corresponding 3D point cloud, The coordinates along the z-axis in the depth coordinate system describe the position of the 3D point cloud in the depth direction (generally understood as the direction perpendicular to the imaging plane in space, pointing towards the depth of the energy-consuming device). The three coordinates of the 3D point cloud can accurately describe the specific orientation of the energy-consuming device in three-dimensional space. In practical implementation, the 3D point cloud can be regarded as the geometric representation of depth data in three-dimensional space. Depth data provides distance information, allowing the extension from the two-dimensional image plane to three-dimensional space, constructing a 3D point cloud that can describe the specific orientation of the energy-consuming device in three-dimensional space; b. Solving the initial transformation parameters using OpenCV and It is understandable that the initial transformation parameters refer to the initial rotation and translation parameters used to describe the coordinate system transformation during the process of aligning and registering visual data with depth data. These parameters are obtained by acquiring the pose pairs of the RGB camera and depth sensor relative to the calibration board. r and t represent the rotation and translation parameters in three-dimensional space, respectively, used to describe the rotation angle and position movement of the camera during the coordinate system transformation. The purpose is to minimize the sum of squared Euclidean distances of all matching point pairs, achieve the initial alignment of visual data and depth data, thereby quickly establishing the initial transformation relationship between the two coordinate systems, reducing the overall offset of the 3D point cloud, providing reasonable initial values ​​for subsequent high-precision optimization, and avoiding iterative optimization from getting trapped in local optima. c. Starting with the initial transformation parameters, using the IterativeClosestPoint(ICP) algorithm from the PCL library, the rotation parameters are continuously adjusted. The resulting rotation matrix R and translation parameters To minimize the objective function To optimize the objective, R and t are continuously adjusted to make the 3D point cloud... With visual feature points The distance error between them gradually decreases until the set convergence conditions are met, such as when the error threshold is less than a certain set value (e.g., 0.1 cm) or when the maximum number of iterations is reached (e.g., 50 times). At this point, the optimal transformation is solved to ensure the spatial consistency between the depth data and the visual data.

[0020] In one embodiment of the present invention, the construction of a three-dimensional point cloud model is achieved through the following steps: S2-1, Feature Extraction and Tracking: The ORB feature detector is used to extract FAST corner points in image frame It, and the set of visual feature points is obtained by traversing all pixels of the image. ,in Using pixel coordinates, and combining optical flow constraint equations for consecutive frames , Calculate feature points Cross-frame feature trajectory, and minimize grayscale error through iteration. Solve for the displacement vector of the feature point. To achieve cross-frame tracking, the average tracking length of the trajectory is set to be no less than 20 frames. Meanwhile, at the image frame interval Internally, depth data is analyzed using the median integral method. Pre-integration optimizes the accuracy of depth data, reduces noise, minimizes error accumulation in the data, and yields the pose changes of energy-consuming devices in the physical scene, including rotational changes. speed change and position changes ; S2-2. Construct an error function to minimize reprojection error and depth data pre-integration error, thereby optimizing the parameters of the energy-consuming device pose and 3D point cloud: In the formula, The state vector contains the camera-to-world coordinate system transformation matrix at all times and the coordinates of all 3D point clouds. For the Cauchy robust kernel function, 3D point cloud coordinates For visual reprojection error, it represents minimizing the 3D point cloud coordinates. After being projected by the camera, and the feature points distance, K is the intrinsic parameter matrix of the RGB camera. for Coordinates in the camera coordinate system These are the intrinsic parameters of the intrinsic parameter matrix after distortion correction in step S1. For reprojection error, The depth data residual term includes pre-integration errors of rotation, velocity, and position, which are solved iteratively using the LM algorithm. The transformation matrix from the camera to the world coordinate system at time t (including rotation) Translation This transformation matrix can convert acquired image data (such as visual feature points) and depth data (such as 3D point clouds) to a unified world coordinate system, enabling the fusion and alignment of multi-source data. and To optimize the speed and position, This represents the x-component of the camera's translation vector in the world coordinate system, that is, the amount of translation of the camera along the x-axis. This represents the z-component of the camera's translation vector in the world coordinate system, i.e., the amount of translation of the camera along the z-axis.

[0021] S2-3, Dense Point Cloud Generation: Utilizing the Optimized Pose With depth data For image pixels (U is the set of effective pixels) Inverse projection generates dense point clouds , Generate dense point clouds, Let K be the inverse matrix of the intrinsic parameter matrix. This represents the depth value at pixel u at time t. Represents the homogeneous coordinate form of pixel coordinate u. Representing pixel coordinates, typically a two-dimensional vector. .

[0022] S2-4, Point Cloud Filtering and Map Building: Setting Voxel Size For dense point clouds Downsampling is performed, retaining the geometric center or average point within each voxel. Statistical filtering is then used to calculate the average distance from each point to its 50 nearest neighbors. , Distance removal Mean +2 standard deviations point, Indicates dense point clouds The i-th point and the j-th point The vector difference between them is used to fuse dense point clouds from multiple frames. Construct point cloud density Dynamic global map M, M= , Represents point cloud data for all time points t. Perform a union operation on the sets to form a 3D point cloud model.

[0023] like Figure 2 As shown, in one embodiment of the present invention, the existing technology has the following defects in the energy consumption category segmentation of 3D models: single-modal segmentation models (such as those based solely on RGB images) do not integrate 3D geometric features, resulting in misalignment between semantic labels and spatial positions, and failure in occluded scene segmentation; at the same time, current multi-source data use offline post-processing fusion, which suffers from time synchronization loss and computational redundancy, failing to meet the real-time needs of industry and making it difficult to adapt to the addition of new equipment or changes in pose, resulting in the decoupling of semantic labels from point cloud positions, directly affecting the reliability of the association between the semantic information and spatial position of the 3D model, and restricting the accuracy of industrial energy consumption analysis. Therefore, the present invention further proposes step S3: S3-1 employs a deep learning model to perform real-time energy consumption category segmentation on RGB images (visual data) collected by AR devices. This identifies and labels key elements in the scene, such as basic building elements and energy-consuming equipment elements, thereby associating equipment parameters with spatial location information and enhancing the model's interpretability. Energy consumption category segmentation is a method of segmenting visual data to identify regions in an image that fall into different energy consumption categories. For example, regions in an image can be divided into "high-energy-consuming equipment areas," "medium-energy-consuming equipment areas," and "low-energy-consuming equipment areas." This provides coarse-grained identification of devices with different energy consumption levels in the scene, helping to quickly locate and identify which areas contain key equipment affecting overall energy consumption. Device instance segmentation, building upon energy consumption category segmentation, further distinguishes different individual devices within the same energy consumption category. For example, in the "high-energy-consuming equipment area," each air conditioner and each CNC machine tool can be identified individually. This segmentation method provides fine-grained identification results, allowing each specific energy-consuming device to be located and tracked individually, providing a foundation for subsequent accurate energy consumption analysis and management.

[0024] Based on the above technical concept, in specific implementation, it is preferable to use a parallel dual-model approach of DeepLabv3+ network and MaskR-CNN to perform pixel-level category labeling and device instance segmentation on the RGB images acquired by the AR device, outputting instance masks and device instance segments to provide visual feature primitives for subsequent spatial alignment. This is achieved through the following steps: Energy-intensive category segmentation aims to classify each pixel in an image, assigning it a specific category label, and segmenting the image into regions with different category meanings, thereby enabling the 3D point cloud model to understand the image content. In practice, a DeepLabv3+ network is used to classify the input image and output a category mask. ,in, Assigning scene element categories to corresponding building elements and energy-consuming equipment, such as labeling the point cloud of a CNC machine tool as "CNC machining equipment," an industrial robot as "automated assembly unit," and a material conveyor belt as "logistics transmission system," thereby achieving pixel-level category differentiation and recognition of equipment in three-dimensional space. Then, the mask is applied... Isolated noise points are removed by morphological opening operation (kernel size 3×3) while preserving the complete outline of scene elements; Device instance segmentation aims to accurately separate each individual object in an image from the background and other objects, thus marking the boundaries and extent of each instance. In practice, the Mask R-CNN model is used to generate candidate bounding boxes to define the areas where energy-consuming devices are located in the physical scene. Then, fixed-size features are extracted from these candidate bounding boxes using the ROIAlign layer, and classification (e.g., device type, such as air conditioner, lighting, etc.) and mask prediction are performed in parallel to obtain the instance mask. , Let H be the feature map (instance) of the i-th feature variable, W be the feature map height, W be the feature map width, and N be the number of feature map channels / number of instances. For the instance mask... The i-th instance has a corresponding binary mask channel of ∈{0,1} H×W This is to accurately identify the pixel position occupied by the instance; S3-2. Use step S1 to correct the intrinsic parameter matrix K and the optimized pose in step S2. With depth data For the obtained category mask (Category mask assigns a specific category label to each pixel (such as "high-power device area", "medium-power device area", and "low-power device area") and instance mask) (Instance mask marks the device instance to which each pixel in the image belongs, such as air conditioner, CNC machine tool) Covering each valid pixel Combined with its corresponding depth value Through three-dimensional spatial geometric transformation, it is converted into 3D coordinates in the camera coordinate system, and then the transformation matrix from camera to world coordinate system is optimized through step S2. Converting these points to world coordinates allows for the mapping of category and instance information to a 3D point cloud, resulting in a 3D point cloud set for that instance. , , Let K be the inverse matrix of the intrinsic parameter matrix. The pixel coordinate u is represented in homogeneous coordinate form. Then, the centroid of the point cloud is calculated using a weighted average. , serving as the spatial coordinates of energy-consuming equipment, where, Here, m represents the 3D point cloud coordinates in the world coordinate system, m is the number of points in the instance 3D point cloud, and m represents the mask. The total number of pixels in the image.

[0025] Example: Continuing with the example of an automobile manufacturing workshop, based on the concepts in S3-1 and S3-2 above, firstly, energy consumption category segmentation is performed on the RGB image of the workshop to identify the pixel regions corresponding to different categories of equipment, such as air conditioners and lighting. However, the category mask can only provide pixel-level category information and cannot distinguish different instances within the same category. For example, if there are two CNC machine tools, the category mask can only identify the category "CNC machine tool," but cannot distinguish between "CNC machine tool 1" and "CNC machine tool 2." This is insufficient for energy consumption prediction and management because different instances have different energy consumption characteristics and operating states. Therefore, based on energy consumption category segmentation, further distinctions are made between different individual devices within the same energy consumption category. For example, in the "high-energy-consuming equipment area," each specific CNC machine tool is identified separately and marked as a different instance (e.g., "Instance 1," "Instance 2") to provide fine-grained identification results, rather than lumping all devices of the same category together. Next, the centroid coordinates of each instance mask are calculated. Assuming that the air conditioner mask of Instance 1 has multiple pixels, the average of the coordinates of these pixels is the centroid coordinates of that instance. Then, depth data fed back from the depth sensor and camera pose obtained through optimization algorithms are acquired. Using the corrected intrinsic parameter matrix and this data, the effective pixels covered by the category mask and instance mask are determined. The corresponding depth values ​​are converted into 3D coordinates in the camera coordinate system. Finally, using an optimized transformation matrix, the 3D points in the camera coordinate system are transformed into the world coordinate system, resulting in the final 3D point cloud. At this point, these 3D point clouds achieve the mapping of category and instance information in space.

[0026] S3-3, Based on the obtained point cloud and center of mass The process of creating database entries for energy-consuming devices is as follows: First, the point cloud set of the device Iterate through all points and find the minimum and maximum coordinate values ​​of each point in the x, y, and z dimensions. These six values ​​together constitute a spatial boundary box describing the spatial range of energy-consuming devices, thereby defining the location and range of instances in space. Then, the instance's category label, such as "CNC machine tool", unique ID, such as "AirCondition_001", centroid coordinates, and spatial bounding box parameters are stored in the node, generating a node attribute structure as shown in Table 1 below. This constructs an energy consumption simulation system, creating detailed database entries for each device. Table 1:

[0027] It is understandable that the 3D point cloud set based on step S3-2... A graph database containing spatial topology and data flow associations is constructed, combining the device category labels, instance types and their corresponding 3D point clouds in the world coordinate system to create detailed database entries for each device. At this point, the spatial location, category mask and instance information of the device are integrated and stored, thereby providing a structured data foundation for subsequent access to more device-related data and energy consumption analysis.

[0028] In one embodiment of the present invention, a multi-threaded IoT protocol interface is used to collect various types of device operation data in a physical scene in parallel using the multimodal sensor array of the AR device. Device operation data (electrical parameters, temperature parameters, flow parameters, and device status parameters) are collected at different frequencies and in different ways to ensure real-time data transmission. The device operation data is then fused with a 3D point cloud model. Through timestamps and spatial location information, various parameter data are associated with corresponding positions in the 3D point cloud model, forming a full-scene digital twin with point cloud as its framework and multi-source parameters as its attributes. This provides a unified spatiotemporal reference for energy consumption analysis.

[0029] In practice, it is preferable to use an extended Kalman filter framework to fuse the multi-source data mentioned above, ensuring data consistency and accuracy. The specific steps are as follows: a. Perform sensor data preprocessing: First, perform spatiotemporal registration on the data collected by the multimodal sensor array (such as light, temperature, and humidity sensors), and synchronize the clocks of all sensors using the NTP protocol to ensure consistency of the time base; second, since the sensor sampling frequencies and coordinate distributions vary, bilinear interpolation is also needed to map the discrete sensor measurements to a unified three-dimensional grid coordinate system: for the sensor data set The measured value of each sensor i Based on its corresponding spatiotemporal coordinates ,in For the physical location of the sensor, For the sampling time, a bilinear interpolation algorithm is used to map discrete measurements to grid nodes; then, synchronous data is filtered using a time window filter (window length ∆T), taking into account the sensor's maximum sampling period and data transmission delay for the sensor data set. Iterate through all data records, taking the sampling time of each data record as an example. Based on the time interval, extract Data from other sensors is incorporated into a temporary dataset, and the dimensions are constructed as follows: The spatiotemporal correlation matrix (where n is the number of sensors and m is the number of data records after time synchronization) is defined, and the matrix elements represent the measurement value of the i-th sensor at the j-th synchronization time point. When there is no valid data, NaN is filled in, and outliers are detected and repaired based on a statistical threshold method. b. Achieve effective fusion of multi-source data: First, taking the data collected by the above-mentioned light, temperature, and humidity sensors as an example, construct a point cloud centroid containing light intensity L, temperature T, humidity H, and the acquired device data. The state vector X, Secondly, the prediction step is executed. Using the state transition matrix F, and based on the temporal correlation of environmental parameters (such as temperature gradient changes), the formula is used... Derive the prior estimate of the state. In the formula, Given the posterior state of the previous time step, derive the prior estimate of the state; then, perform an update step, using the light intensity values ​​measured by the sensor. ,temperature ,humidity and centroid measurement value observation vector Through Kalman gain In the formula, Let H be the prior covariance, H be the observation matrix, and R be the observation noise covariance. Combined with the formula... The state estimates are corrected to achieve effective fusion of multi-source data (environmental data and centroid spatial data), generating a structured dataset that includes spatial, equipment, environmental, and temporal data. This allows for a more comprehensive reflection of the physical scene information. Based on the energy consumption simulation system, as shown in Table 2 below, the database entries for devices within the physical scene have been created. Table 2:

[0030] It should be noted that the state transition matrix F needs to be constructed based on environmental parameters and the physical laws of the change of the centroid over time (such as the evolution model of temperature over time, the movement law of the equipment centroid due to production activities, etc.). The observation matrix H is used to describe the mapping relationship from the state vector to the observation vector. If each parameter can be directly measured and corresponds, then H can be set as an identity matrix.

[0031] Based on the above technical concept, it should be noted that in step S3-1, after energy consumption category segmentation and device instance segmentation, preliminary category masks and instance masks are obtained. However, this preliminary segmentation result has the following problems: In the energy consumption category segmentation results, there are noise points and blurred boundaries. For example, in complex industrial scenarios, similar-looking equipment (such as CNC machine tools and ordinary machine tools) may be incorrectly classified in some areas due to similar shapes and colors, which will reduce the accuracy of energy consumption category segmentation. Secondly, in the equipment instance segmentation results, there are also cases of inaccurate contours, missed detections, or false detections. Small equipment arranged closely together may be mistakenly merged into one instance, while large equipment may be incorrectly segmented into multiple instances due to occlusion or deformation, thereby reducing the accuracy of equipment instance segmentation. The direct result is that the initial segmentation incorrectly associates instances with the wrong category mask (e.g., misclassifying a "CNC machine tool" instance as a "general machine tool"), or omits the association between instances and the category mask (e.g., failing to identify some equipment instances belonging to the "energy-consuming equipment" category), weakening the relevance of the segmentation results. Finally, if these initial segmentation results with biases, noise, and inaccurate associations are directly used for subsequent spatial alignment, it will lead to inaccurate spatial alignment, thereby affecting the association of equipment parameters and the performance of subsequent functional modules, ultimately reducing the accuracy of the system's analysis and processing of information such as equipment energy consumption and operating status.

[0032] like Figure 2 As shown, based on this, the initial category mask is output. and instance mask Furthermore, it is necessary to process the following segmentation optimization mechanism to continuously evolve the deep learning model, reduce the subsequent segmentation error rate, and thus provide high-precision, strongly correlated visual feature primitives for subsequent spatial alignment.

[0033] The process is as follows: S3-11, Output category mask As a global feature representation, it is used to represent the image. Overall, coarse-grained category structure information, expressed as an output instance mask. Local feature representation, used to represent images Specific energy-consuming devices and fine-grained instance information are used to construct hierarchical relationships: the fine-grained category label (class) of each instance i is... i Mapped to a category of a predefined overall structure In Chinese, for example, the class "CNC machine tool" i Mapped to "energy-consuming device" c, denoted as: c = M(class i ), where M is a predefined mapping function.

[0034] S3-12, For each instance i detected by the MaskR-CNN model (including the instance mask) Category tags i , Mapping class mask ), calculate its relationship with the class mask Consistency score in region C Region C is the category mask. The pixel region belonging to class c is considered as the set of all pixels predicted to be class c in the class mask. In the following formula, this region C is represented by a binary mask image. This indicates that the consistency score measures the agreement between the predicted category and the expected category c within an instance region. The calculation process includes: First, let's start with the category mask. Generate category mask binary mask : , Second, calculate the instance mask. With category mask binary mask The intersection-union ratio is used to obtain the consistency score. This measures the degree of overlap between the segmented region of a device instance and the corresponding category mask region. In the formula, For logical AND operation, it represents the category mask. and The intersection of the two masks represents the overlapping region C containing the instance mask and the class mask. The logical OR operation represents the category mask. and binary mask The union of the instance mask and the class mask represents the total coverage area of ​​region C. This indicates the segmented region of the device instance and the corresponding category mask region. The degree of overlap of coverage, when When =1, it represents an instance mask. Complete with category mask region overlap, when When =0, it represents an instance mask. Complete with category mask region They do not overlap at all.

[0035] Example: Assuming an instance mask With category mask area The two binary masks are shown below: The size is 3×3, and the pixel positions with a value of 1 are (1,1), (1,2), (1,3), (2,1), (2,2), (3,1). Given a 3x3 matrix, where the pixel positions with a value of 1 are (1,1), (1,2), (2,1), (2,2), (3,1), (3,2), we get... The overlapping pixel positions are (1,1), (1,2), (2,1), (2,2), (3,1), and the number of pixels in the determined intersection is 5. Similarly, we get The covered pixel positions are (1,1), (1,2), (1,3), (2,1), (2,2), (3,1), (3,2), and the number of pixels in the determined union is 7. The final obtained consistency score = 5 / 7 ≈ 0.714, and we get the example mask Overlapping with the category mask region The overlapping degree is 71.4%.

[0036] S3-13. Set the consistency threshold Tcon, such as Tcon = 0.8. If the calculated , then the segmentation result of this instance is considered reliable, and this instance is retained; On the contrary, it is determined that there are potential errors or ambiguities in the segmentation result of instance i. For example, the instance is segmented into the wrong category region, the category prediction is incorrect, or the mask is inaccurate. At this time, based on the consistency score , through the formula , an initial confidence weight W is assigned to each instance i i , where k is a scaling factor used to control the steepness of the function, and W i ∈(0,1). Generally speaking, when S i ≥ Tcon, take W i ≈ 1, indicating that the confidence of the instance is relatively high and it is included in the candidate correct instance set. And the following process is triggered: Secondary verification: For instances with S i < Tcon, use the temporal information to perform secondary verification on the stability of the instance trajectories in consecutive frames . If all the information of the low-confidence instances to be verified in the current batch has been verified (the scores are all higher than the threshold), then mark them as reliable instances, add them to the candidate correct instance set and terminate the optimization mechanism. Otherwise, mark the information of the un-verified instances as the Uncertain state for manual review and processing; after completing the processing of the instances in the current batch, the optimization mechanism terminates.

[0037] Based on the above technical concept, it can be understood that the purpose of W i is to quantify the preliminary reliability evaluation based only on the visual consistency of the current frame: W i close to 1 indicates less visual ambiguity, and W i close to 0 indicates extremely poor visual consistency, providing priorities or initial states for subsequent processing (especially secondary verification). It should be noted that in practical applications, to optimize computing resources, the quantification of Si "much less than" Tcon can be achieved by setting a significant deviation threshold, and take W i≈0, indicating that the confidence of the instance is low, and it is directly included in the set of mis-segmented instances, W i Instances close to 0 are directly regarded as errors and do not require verification or are of low-priority verification; while W i Instances close to Tcon are more worthy of verification. Subsequently, for these S i Instances < Tcon trigger a secondary verification process based on the stability of temporal energy consumption (analyzing the power mean, variance, and correlation coefficient ρ with time in consecutive frames, etc.). If the instance passes the secondary verification (stable energy consumption, meeting the preset criteria), its confidence weight is updated (formula 10 below), and the instance is marked as reliable and added to the set of candidate correct instances; if the verification fails or the energy consumption is abnormal, the instance is marked as in the Uncertain state and awaits manual review.

[0038] At this point, the finally output structured segmentation result integrates the precise boundary mask instance masks of high-confidence instances and the fine-grained category label class i , and also annotates the category c to which each instance belongs and its confidence weight , and retains the overall energy consumption category segmentation result , thereby enhancing the context relevance of the segmentation result and providing high-precision and strongly category-associated visual feature primitives for subsequent spatial alignment and device parameter association.

[0039] In an embodiment of the present invention, the process of using temporal information for secondary verification is as follows: First, for a time window such as (corresponding to 10 frames, frame rate 5fps), traverse the consecutive frame sequence , collect the temporal energy consumption data of the device. For ∈[t−Δt,t+Δt], based on the sensor array, obtain the real-time power of the device at time as , and obtain the temporal energy consumption data of the energy consumption device within the consecutive time window; Secondly, calculate the mean energy consumption (power) and variance of the energy consumption device within the time window to measure the stability and fluctuation of the device's energy consumption: , , where is a time series variable representing each time point within the traversed time window; Thirdly, calculate the correlation coefficient between the temporal energy consumption and the time series to verify the stability of the device's energy consumption over time, thereby determining whether the device instance segmentation result is reliable: , where The value ranges from [0,1] and measures the linear correlation between energy consumption and time. The mean of the time series; Therefore, a consistency judgment is made: through quantitative analysis. Mean correlation coefficient with similar historical examples , correlation coefficient standard deviation of similar historical examples Jointly determine the relationship: When satisfied (k is an adjustable hyperparameter; a value of k=1.5 is recommended, covering approximately 87% of normally distributed data), and , To preset an energy consumption fluctuation threshold for determining whether energy consumption changes are abnormal, an instance confidence weight update operation is performed according to the following formula. At this point, the device energy consumption is considered stable, indicating that the device instance segmentation result is reliable. Verification shows that the device instance segmentation result has stability and accuracy in continuous video streams. Simultaneously, instances meeting the above conditions are included in the candidate correct instance set. Store data in the form of triples: entity, relation, category, such as CNC machine tool, belonging to energy-consuming equipment. In the formula, For the equipment in Real-time power at any given moment For the equipment in Real-time power at any given moment The updated confidence weight for instance i.

[0040] If the correlation coefficient Exceeding the historical data distribution range ( Or abnormal energy consumption fluctuations may occur. If the instance segmentation result is incorrect, the instance is marked as "Uncertain" and a manual review process is triggered. Instances confirmed by manual review are then added to the erroneous segmentation instance set. In the form of quintuples , Describe the error type, such as a safety identifier being incorrectly mapped to an energy-consuming device: =(Security Identification Mask, Energy Consumption Device Area, Security Identification, Non-Energy Consumption Device, Category Mismapping), or lighting equipment is incorrectly mapped as a high-energy consumption device. =(Lighting equipment mask, high energy consumption equipment area, lighting equipment, medium energy consumption equipment, misclassification of energy consumption level), store it, and if the number of instances that need to be manually reviewed every day drops to less than 20% of the initial amount, then stop model optimization or reduce the optimization frequency.

[0041] like Figure 3As shown, in one embodiment of the present invention, to further improve the accuracy and reliability of the model in energy consumption prediction, a daily calibration mechanism based on incremental learning is also needed. This mechanism injects the knowledge of the aforementioned sets of correct and incorrect instances into the loss function, dynamically adjusting the model parameters. Specifically, a repulsive penalty is applied to incorrect instances, while an attractive penalty is applied to correct instances, thereby optimizing the model parameters until the model's error rate significantly decreases, thus improving the accuracy of subsequent energy consumption predictions. The process is as follows: First, from the correct set of instances Extract all correct instances stored in triplet form (e.g., ⟨CNC machine tool, belonging to, energy-consuming equipment>), and generate entity feature embedding vectors using the ResNet-50 backbone network of Mask R-CNN. Simultaneously, from the incorrectly segmented instance set Obtain incorrect segmentation cases in quintuple format (e.g., ( , (Safety identifiers, non-energy-consuming devices, category mismapping) are corrected by generating target embedding vectors using the ASPP module in DeepLabv3+. Fusion equipment operating parameters (real-time power) Correlation coefficient To construct the daily incremental training dataset ; Secondly, through the multimodal loss function framework Continuous optimization of model updates, including Basic segmentation loss function Used to measure the model's performance in pixel classification tasks, the cross-entropy loss function optimizes the model's classification accuracy for each pixel: In the formula, N is the total number of pixels, that is, the number of pixel samples involved in the calculation. This is the true label of the i-th pixel (1 if it belongs to the target category, 0 otherwise), used to indicate the actual classification of that pixel. The model predicts the probability of the i-th pixel, representing the confidence level of the model in judging that the pixel belongs to the target category. When the model prediction differs greatly from the true label, the loss value increases significantly. Error suppression loss function This is used to correct prediction bias in the model on erroneous instances by penalizing the distance between erroneous feature embeddings, reducing repeated outputs of erroneous patterns by the model. For example, it forces the model to move the "safety label" entity closer to the "non-energy-consuming device" category in the feature space, thus eliminating mismapping. ; Correctly enhance the loss function This is used to enhance the model's ability to identify correct instances by maximizing the similarity between correct feature embedding pairs, thereby increasing the model's confidence in correct patterns. For example, it can enhance the embedding correlation between "CNC machine tool" and "energy-consuming equipment". , It is the sigmoid function; Next, gradient backpropagation is performed to optimize the weights of the ROIAlign layer in Mask R-CNN (to enhance sensitivity to error regions) and the parameters of the ASPP module in DeepLabv3+ (to strengthen the association with correct classes), and an adaptive learning rate is adopted. Execution parameter update , The gradient of the total loss function with respect to the parameters is used to immediately validate the mIoU with unlabeled samples after the update (if the decrease exceeds 5%, roll back and restore the parameters of the previous version) and continue optimization. Otherwise, it indicates that the model performance has stabilized and optimization is stopped to ensure the stability of model updates. Finally, monitor the error recurrence rate. When the error recurrence rate When the recurrence rate is below 10% of the threshold, the preset energy consumption fluctuation threshold is increased to strengthen the screening criteria, and model optimization is stopped. When the error recurrence rate exceeds 10%, the preset energy consumption fluctuation threshold is decreased to relax the screening criteria, and optimization continues. This is the set of error instances detected in the new cycle. This is the set of historical error instances, representing the set of error instances recorded by the model within a historical period.

[0042] In one embodiment of the present invention, step S4 is further proposed to construct and train a ConvLSTM-GNN hybrid model to achieve accurate prediction of energy consumption data. The specific implementation steps include: S4-1, For structured data groups Multi-dimensional feature extraction and analysis are performed to extract key feature vectors that accurately describe energy consumption scenarios, thereby obtaining detailed information on equipment operating status, environmental data, time periodicity, and historical energy consumption trends. The process is as follows: First, time feature processing: timestamps are converted into periodic codes, and trigonometric functions are used to cyclically encode hours, week numbers, and months, incorporating periodic patterns. Simultaneously, Boolean values ​​for holidays are overlaid to distinguish special dates, integrating these into a time feature set. Second, an equipment operating status vector is constructed. To comprehensively characterize equipment operating conditions: One-hot encoding is used to characterize equipment operating modes, denoted as... If an energy-consuming device is in different operating modes such as cooling, heating, or standby, set the corresponding dimension of its encoding vector to 1, and the other dimensions to 0; for example, set the device load rate... Start-stop status (Value 1 indicates running status, 0 indicates stopped status) and temperature setpoint As a numerical feature, it is related to the operating mode encoding vector. Together they form the state vector ,Right now Secondly, standardize the environmental characteristic matrix: such as organizing indoor temperature. ,humidity ,illumination and outdoor temperature ,humidity Wind speed Precipitation probability Forming an environmental feature matrix .

[0043] Z-score standardization is used to eliminate the influence of dimensions. The formula is: In the formula, For the standardized i-th environmental feature, This is the historical average of this feature. The historical standard deviation is used; finally, time, equipment status, and environmental data are integrated to generate key feature vectors. .

[0044] S4-2, ConvLSTM-GNN Model Construction and Training: First, ConvLSTM temporal encoding is performed using 32 3×3 convolutional kernels. The cell state and hidden state are dynamically updated with the help of a gating mechanism to achieve the extraction of temporally dependent features.

[0045] Secondly, a fully connected output and training configuration is performed to construct a 2-layer 64-dimensional GNN, and the features of the GNN output nodes are integrated and mapped to energy consumption prediction values ​​through a fully connected layer. Meanwhile, the Huber loss function is used to balance the prediction error, as shown in the formula: In the formula, The threshold value represents the balance between mean squared error and mean absolute error characteristics. When configuring, the optimizer chosen is AdamW, and the learning rate is... Weight decay To suppress overfitting, the dataset is divided into 70% (training), 15% (validation), and 15% (test). The maximum training cycle is set to 200 rounds, and the batch size is 256. Training is stopped if the validation set loss does not decrease, in order to ensure the model's generalization ability.

[0046] S4-3. Employing a tiered strategy to achieve accurate prediction of equipment energy consumption: For short-term predictions (e.g., 1–4 hours, 5-minute granularity), and to meet high-frequency real-time scheduling requirements, a 5-minute granularity feature vector is used. As input, the ConvLSTM-GNN model is used to capture fine-grained temporal fluctuations, and the output is a 48-step predicted sequence. This enables real-time demand response; For medium- to long-term forecasts (e.g., 1-7 days, hourly granularity), the focus is on equipment maintenance and energy efficiency assessment, generating hourly features by downsampling the original features. By using a 4-layer encoder-decoder and an 8-head self-attention mechanism to mine its long sequence dependencies, it outputs a 168-step prediction sequence, thereby achieving real-time demand response.

[0047] Example: Taking a large air conditioning system in automobile production as an example, this system adjusts its energy consumption every 5 minutes based on changes in the indoor and outdoor environment and personnel activity. To achieve refined energy management, it is necessary to make short-term predictions of energy consumption for the next 4 hours in order to adjust the operating strategy in a timely manner and meet real-time demand response. In this physical scenario, the feature granularity is set to 5 minutes. The energy consumption data every 5 minutes and related influencing factors (such as historical energy consumption, indoor and outdoor temperature, humidity, and personnel flow in the same time period) constitute the feature vector xt. A ConvLSTM-GNN model is used. During implementation, the 5-minute granularity feature vector xt is input into the model to capture fine-grained temporal fluctuations and output the prediction sequence for the next 48 5-minute time steps (i.e., 4 hours). This allows managers to know in advance the energy consumption for each 5-minute period within the next 4 hours, making it easier to adjust the operating parameters of the air conditioning system in a timely manner to cope with changes in real-time demand and achieve energy conservation.

[0048] Based on the above technical concepts, traditional models in equipment energy consumption prediction typically only provide a single-point prediction result, lacking quantification of prediction uncertainty. For example, a simple linear regression model might predict that a device will consume 100 kWh of electricity in the next hour, but it doesn't indicate the reliability of this prediction or whether there is significant uncertainty. The direct problem is that a single prediction result has obvious shortcomings in practical applications: when the prediction deviates significantly from the actual situation, it leads to decision-making errors. For instance, in energy management, failure to accurately predict the fluctuation range of energy consumption can result in insufficient or excessive energy supply, thus affecting the stable operation of the system and the rational utilization of resources.

[0049] Based on this, this invention further proposes S4-4, employing the Monte Carlo dropout method to generate confidence intervals, quantifying the uncertainty in the energy consumption prediction process by combining point predicted values ​​with probability intervals. The process is as follows: First, construct a neural network module with a Dropout layer, and during the fully connected output and training configuration of the above model, set the Dropout probability, and use historical energy consumption data to train and optimize the model; Secondly, during the prediction phase, the Dropout probability remains constant, and multiple (e.g., 100) random forward propagations are performed on the same sample. (Including historical energy consumption data, temperature, humidity, and other features of the device), multiple forward propagations are performed. Because the Dropout layer randomly discards different neurons during each propagation, the output prediction value varies. Since there are differences among (i=1,2,…,100), the statistic and confidence interval are calculated using the following formula: Point Predicted Value Next, the 100 predicted values ​​are sorted, and quantiles are calculated using linear interpolation to obtain confidence intervals at different confidence levels, such as the 95% confidence interval. and 80% confidence interval ; Finally, the predicted point values ​​under this confidence interval As the main result of energy consumption forecasting, it is used for daily management, and the uncertainty of the forecast is quantified based on the confidence interval, so as to provide the credibility of the forecast results for energy consumption decision-making.

[0050] In one embodiment of the present invention, in order to achieve accurate visualization and real-time monitoring of energy consumption data, the system also needs to build a low-latency data pipeline (such as using the stream processing framework Apache Kafka for real-time data processing and analysis, configuring 3 partitions and 3 replicas, and setting a throughput of 10,000 messages / second), receive energy consumption and device status data pushed by the IoT gateway, and perform real-time cleaning and preprocessing. Among them, the Z-score method is preferred for outlier detection, missing values ​​are filled in according to type, and the calculation latency is controlled within 20ms. Finally, the cleaned data is associated with a database based on binding relationships, and an optimization problem model is constructed with minimizing energy consumption cost, total energy consumption, or peak demand as the objective function. The energy consumption device control strategy is output and visualized based on an AR interface.

[0051] In one embodiment of the present invention, as a second aspect of the present invention, a real-time energy consumption simulation and optimization system for all scenarios based on AR devices is proposed, including a memory and a processor, wherein the memory includes a real-time energy consumption simulation and optimization program for all scenarios based on AR devices, and when the real-time energy consumption simulation and optimization program for all scenarios based on AR devices is executed by the processor, the real-time energy consumption simulation and optimization method for all scenarios based on AR devices described above is implemented.

[0052] The technical scope of this invention is not limited to the content described above. Those skilled in the art can make various modifications and variations to the above embodiments without departing from the technical concept of this invention, and all such modifications and variations should fall within the protection scope of this invention.

Claims

1. An AR device-based full-scene real-time energy consumption simulation optimization method, characterized in that: The method comprises the steps of: Collecting visual data, depth data and operation data representing the target energy-consuming device in the physical scene using the multi-modal sensor array of the AR device, and constructing a three-dimensional point cloud model representing the scene where the energy-consuming device is located; Performing energy consumption category segmentation and device instance segmentation on the collected visual data, and constructing a segmentation optimization mechanism to improve the accuracy of the segmentation results and identify the energy-consuming device elements in the physical scene; the execution process of the segmentation optimization mechanism is as follows: Measuring the coverage overlap degree between the instance mask output by the device instance segmentation and the corresponding category binary mask output by the energy consumption category segmentation by calculating the intersection over union between them; setting a consistency threshold, if the coverage overlap degree is higher than the threshold, the instance information is retained and output as the structured segmentation result; Constructing an energy-consuming device database to realize accurate identification of the energy-consuming device in the scene, the execution process is as follows: After spatial alignment of the optimized segmentation results, three-dimensional point cloud model data and operation data, converting them into world coordinate points to realize mapping of category and instance information to 3D point cloud, and obtaining a 3D point cloud set of instance information; secondly, calculating the centroid of the 3D point cloud by weighted average to obtain the positioning coordinates of the target device in space; finally, creating an energy-consuming device database entry according to the obtained point cloud and centroid: traversing the 3D point cloud set to find the minimum and maximum coordinate values of each point cloud in x, y and z dimensions to form a spatial bounding box, defining the location and range of instance information in space, storing the category label, unique ID, centroid coordinates and spatial bounding box parameters of the instance information into a node, and constructing the device database entry of the energy simulation system to realize strong binding of the segmentation results, three-dimensional point cloud model data and device operation data; According to the identified energy-consuming device information, constructing a spatio-temporal sequence model to predict energy consumption, and realizing visualization based on the AR interface according to the energy consumption prediction results. 2.The AR device based full-scene real-time energy consumption simulation optimization method of claim 1, wherein: By computing an instance mask M i and the intersection over union ratio of the category mask M s , a consistency score S i is obtained, which realizes the measurement of the coverage overlap degree, wherein if the coverage overlap degree is lower than a consistency threshold Tcon, it is determined that the segmentation result of the instance i has potential errors or ambiguities, at which time the consistency score S i is configured for each instance i, and a confidence weight W i is triggered, and the following process is triggered: For instance information with potential errors or ambiguities, continue to use the collected time series information containing visual data to analyze continuous frames, and perform secondary verification on the stability of the instance information track in the continuous frames, if all instance information is verified and the score is higher than the threshold, terminate the optimization mechanism, otherwise mark the instance information that has not been verified as Uncertain state for manual review and processing, the execution process of the secondary verification is as follows: First, energy consumption devices based on sensor arrays are... The real-time power at time is First, obtain the time-series energy consumption data of the energy-consuming equipment within a continuous time window; second, calculate the average energy consumption of the energy-consuming equipment within the time window. and variance First, it measures the stability and fluctuation of equipment energy consumption; second, it calculates the correlation coefficient between energy consumption time series and time series. The reliability of the device instance information segmentation results is determined; finally, quantitative analysis is performed. Mean of correlation coefficient with similar historical instances Standard deviation Consistency determination is made regarding the relationship. 3.The AR device based full-scene real-time energy consumption simulation optimization method of claim 2, wherein: The correlation coefficient The calculation formula is: , wherein, is the mean value of the time series, is the time window radius. 4.The AR device based full-scene real-time energy consumption simulation optimization method of claim 2, wherein: The execution process of the consistency determination is as follows: When the conditions are met , and , is a preset energy consumption fluctuation threshold, the instance confidence weight updating operation is performed, at this time, it is considered that the device instance information segmentation result is reliable, and the instance information meeting the above conditions is included in the candidate correct instance set through verification , and stored; If or , it is considered that the device instance information segmentation result is wrong, the instance is marked as the Uncertain state, and the manual review process is triggered. After manual confirmation of the existence of the error instance, it is included in the error segmentation instance set , and stored. 5.The AR device based full-scene real-time energy consumption simulation optimization method of claim 4, wherein: If there is instance information that has not been verified and the score is lower than the preset threshold, or the number of instances confirmed to have errors by manual review reaches the maximum set number, the segmentation model segmentation effect needs to be improved based on the constructed incremental learning mechanism, the process is as follows: First, based on the correct instance set and the error segmentation instance set construct the daily incremental training data set Second, continuously optimize the segmentation model update through the multi-modal loss function framework , wherein the basic segmentation loss function , wherein N is the total number of pixels, is the true label of the i-th pixel, is the prediction probability of the i-th pixel by the model, and when the model prediction differs greatly from the true label, the loss value significantly increases; is the error suppression loss function; is the correct reinforcement loss function; third, perform gradient backpropagation calculation, optimize the ROIAlign layer weight and ASPP module parameters of the segmentation model, and perform parameter update using an adaptive learning rate, and immediately after the update, verify the mIoU using unlabeled samples, continue to optimize, otherwise, it indicates that the model performance has tended to be stable, and stop optimizing to ensure the stability of the model update; finally, monitor the error recurrence rate , when the error recurrence rate is lower than the recurrence rate threshold, the preset energy fluctuation threshold is increased, the screening standard is strengthened, and the model optimization is stopped; when the error recurrence rate exceeds the recurrence rate threshold, the preset energy fluctuation threshold is reduced, the screening standard is relaxed, and the optimization is continued.

6. The AR device-based full-scene real-time energy consumption simulation optimization method according to claim 1, characterized in that: In the process of constructing the energy-consuming device database, multiple data fusion is realized through an extended Kalman filter framework to ensure data consistency and accuracy, the process is as follows: Firstly, sensor data preprocessing is performed, the data collected by the multi-modal sensor array is spatio-temporally registered, and the clocks of all sensors are synchronized by using the NTP protocol to ensure the consistency of the time reference; secondly, the sensor discrete measurement values are mapped to a unified three-dimensional grid coordinate system by bilinear interpolation, and the synchronous data is filtered by using a time window filter to construct a spatio-temporal correlation matrix, detect and repair the abnormal values of the sensor data; thirdly, a state vector containing the operating data of the energy-consuming equipment and the 3D point cloud of the centroid of the state is constructed, a prediction step is performed, the prior estimate value of the state is derived by using a state transition matrix, and an update step is performed to correct the state estimate value by combining the observation vector through the Kalman gain to realize the effective fusion of multi-source data; finally, a structured data group is finally generated, and the database entry creation about the equipment in the physical scene is completed.

7. The AR device-based full-scene real-time energy consumption simulation optimization method according to claim 1 or 6, characterized in that: The process of constructing a spatio-temporal sequence model to predict energy consumption is as follows: Firstly, feature extraction is performed on the structured data set to obtain a key feature vector representing the energy-consuming equipment in the physical scene; secondly, a ConvLSTM-GNN model is used to perform time series coding on the key feature vector through ConvLSTM to extract time-dependent features representing the time correlation and dynamic change rule of the key feature vector, and then integrate the GNN output node features through full connection output and training configuration, and map them to energy consumption prediction values through a full connection layer; thirdly, a Huber loss function is used to balance the prediction error, and an AdamW optimizer is used for training to ensure the generalization of the model; finally, a hierarchical strategy is constructed to realize accurate prediction of device energy consumption. 8.The AR device based full-scene real-time energy consumption simulation optimization method of claim 7, wherein: In the process of energy consumption prediction using the spatio-temporal sequence model, a Monte Carlo dropout method is also used to generate a confidence interval in the form of point prediction value combined with probability interval to quantify the uncertainty in the energy consumption prediction process, and the process is as follows: During the training process, the dropout probability is set to prevent overfitting. In the prediction stage, the dropout probability is kept unchanged, and the point prediction value is calculated by formula . Then, the N prediction values are sorted. The quantile is calculated by linear interpolation, and the confidence interval under different confidence levels is obtained. Finally, the point prediction value under this confidence interval is taken as the main result of energy consumption prediction. ​ 9.The AR device based full-scene real-time energy consumption simulation optimization method of claim 1, wherein: The collected data is standardized and preprocessed according to the hierarchical calibration mechanism, and the process is as follows: Firstly, distortion correction is performed on the visual data: based on Zhang's calibration model, multiple images at different angles are captured, corner coordinates are detected using the cornerDetect function of OpenCV, and the least squares method is used to calculate the intrinsic matrix and distortion coefficient vector of the RGB camera to correct the radial and / or tangential distortion of the image, reduce the image edge, and improve the shape and position measurement accuracy of the target device; secondly, accurate registration of depth data and visual data is achieved: the focal length in the intrinsic matrix K and the image principal point coordinates are used to project the pixel coordinates representing the image position and the depth values representing the distance information of the pixel points to the RGB camera into 3D point clouds, and invalid depth points are removed to generate a matching point pair set; Thirdly, the initial transformation parameters are solved by OpenCV to achieve the preliminary alignment of visual data and depth data, and to reduce the overall offset of the 3D point cloud; finally, the ICP algorithm of the PCL library is used to solve the optimal transformation by constructing a minimization objective function to gradually reduce the distance error between the 3D point cloud and the visual feature points until the convergence condition is met, achieving the spatial consistency of the depth data and the visual data.

10. The AR device based full-scene real-time energy consumption simulation optimization system of claim 1, wherein: A memory and a processor, wherein the memory includes an AR device-based full-scene real-time energy consumption simulation optimization program, and the AR device-based full-scene real-time energy consumption simulation optimization program, when executed by the processor, implements the full-scene real-time energy consumption simulation optimization method of claims 1-9.

Images