AR device-based full-scene real-time energy consumption simulation optimization method and system

Abstract

The application provides an AR device-based full-scene real-time energy consumption simulation optimization method and system, comprising the following steps: collecting visual data and depth-of-field data of a physical scene; fusing the preprocessed visual data and depth-of-field data to construct a three-dimensional point cloud model; performing energy consumption category segmentation and device instance segmentation on the collected visual data, and constructing a segmentation optimization mechanism to improve the accuracy of the segmentation result; constructing an energy consumption device database to realize accurate identification of energy consumption devices in the scene; and realizing visualization based on an AR interface. The application adopts a double-model parallel architecture, performs pixel-level category labeling on input images, outputs category masks, divides the images into regions with different category meanings, assigns specific category labels to each pixel, improves the device identification accuracy, ensures strong binding of the device category labels and the point cloud positions during subsequent spatial alignment, and provides reliable device information for energy consumption analysis.

Description

Technical Field

[0001] This invention relates to the field of energy consumption simulation technology, specifically to a method and system for real-time energy consumption simulation and optimization across all scenarios based on AR devices. Background Technology

[0002] With the increasing demands for energy efficiency in the industrial and building sectors, energy consumption monitoring and optimization systems are playing an increasingly important role in practical applications.

[0003] Currently, regarding database construction, patent CN118247781A proposes a deep learning-based industrial robot target recognition method and system. It aims to construct an industrial robot recognition model using 3D point cloud data acquired by deep learning technology and depth cameras, solving the problem of target recognition in complex scenarios. However, it has several significant shortcomings in the field of industrial robot target recognition. ① In terms of data acquisition, relying solely on depth cameras to obtain point cloud data easily leads to misjudgments in practical applications. When facing motors / other equipment of different models but similar shapes, it is difficult to distinguish them based solely on point cloud data, significantly increasing the misjudgment rate and affecting data accuracy. ② In the model building stage, its cascaded model architecture exposes the problem of losing detailed features during data transmission. Data transmission from the semantic model to the target discrimination model requires downsampling (Ns×3→Ns×32), resulting in the loss of detailed features, such as object texture features, affecting the accuracy of the discrimination model and consequently impacting the real-time performance and accuracy of the entire system. ③ In the scenario database construction stage, the semantic granularity is insufficient, and the distinction between equipment and non-equipment entities is not effectively made. This results in a lack of accurate identification and classification of specific equipment entities in the data, leading to errors due to inaccurate or untimely data updates. When the physical attributes of the equipment change (such as equipment upgrades or replacement of parts), the database cannot reflect these changes in a timely manner, causing the analysis results based on the database to be inconsistent with the actual situation, affecting the accuracy and reliability of the data.

[0004] In the area of ​​energy consumption management, CN119167792A proposes an adaptive calculation method, system, equipment, and medium for building heating and cooling loads. Through computer vision and machine learning technologies, combined with building information modeling, it solves the data acquisition and accuracy problems of traditional methods, achieving high-precision and adaptive calculation of building heating and cooling loads, thus improving the efficiency of energy management and supporting sustainable building development. However, it has the following logical flaws in the field of building heating and cooling load prediction:

[0005] ① Inadequate post-segmentation processing: After extracting image features using edge detection, feature point recognition, and semantic segmentation techniques, the segmented data was not further validated or optimized for subsequent model improvements. For example, using the Mask R-CNN model to semantically segment the image data of air conditioning equipment inside a building, the location and shape of the equipment could be identified. However, without further validation and optimization of the segmentation results, the segmented data suffered from blurred boundaries, misidentification, and size deviations. This resulted in the segmented air conditioning equipment area including pixels of non-air conditioning equipment or omitting pixels of some air conditioning equipment, affecting the accuracy of the overall building's heating and cooling load prediction. ② The energy consumption prediction model primarily relied on random forest and neural network algorithms, combined with environmental parameters such as electricity consumption data, equipment power, and solar radiation. However, this approach did not adequately consider how to effectively integrate multimodal data or how to quantify the uncertainties in the prediction process within the model. ③ Finally, in the data fusion and preprocessing stage, the building's image data and electricity consumption data were preprocessed to extract time-series features. However, when performing 3D reconstruction and BIM model generation, these time-series features were not considered to enhance the model's ability to capture dynamic changes in building energy consumption.

[0006] Therefore, there is an urgent need for a real-time energy consumption simulation and optimization system that can achieve unified perception and fusion of spatial location, equipment status and environmental data across all scenarios, in order to support more accurate and comprehensive energy consumption analysis and optimization. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for real-time energy consumption simulation and optimization across all scenarios based on AR devices, thereby resolving the problems mentioned in the background section.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a real-time energy consumption simulation and optimization method for all scenarios based on AR devices, comprising the following steps:

[0009] By utilizing the multimodal sensor array of AR devices, visual data, depth data, and operational data representing target energy-consuming devices in physical scenes are collected to construct a three-dimensional point cloud model representing the scene where the energy-consuming devices are located.

[0010] The collected visual data is segmented by energy consumption category and device instance. A segmentation optimization mechanism is constructed to improve the accuracy of the segmentation results and identify energy-consuming device elements in the physical scene. The execution process of the segmentation optimization mechanism is as follows:

[0011] The overlap ratio between the instance mask output by device instance segmentation and the corresponding category binary mask output by energy consumption category segmentation is used to measure the degree of coverage overlap between the region where the device instance segmentation is located and the region where the corresponding category mask is located. A consistency threshold is set. If the degree of coverage overlap is higher than the threshold, the instance information is retained and output as the structured segmentation result.

[0012] The process of building an energy-consuming device database to accurately identify energy-consuming devices in a scenario is as follows:

[0013] After spatial alignment of the optimized segmentation results, 3D point cloud model data, and operational data, they are converted into world coordinate system points to map category and instance information to the 3D point cloud, resulting in a 3D point cloud set of instance information. Next, the centroid of the 3D point cloud is calculated using a weighted average and used as the spatial coordinates of the target device. Finally, based on the obtained point cloud and centroid, energy consumption device database entries are created: the 3D point cloud set is traversed, and the minimum and maximum coordinate values ​​of each point cloud in the x, y, and z dimensions are found to form a spatial bounding box, defining the location and range of instance information in space. The category label, unique ID, centroid coordinates, and spatial bounding box parameters of the instance information are stored in the node, constructing the device database entries of the energy consumption simulation system, achieving strong binding between the segmentation results, 3D point cloud model data, and device operational data.

[0014] Based on the identified energy-consuming equipment information, a spatiotemporal sequence model is constructed to predict energy consumption, and the energy consumption prediction results are visualized using an AR interface.

[0015] As a second aspect of the present invention, a real-time energy consumption simulation and optimization system for all scenarios based on AR devices is proposed, including a memory and a processor. The memory includes a real-time energy consumption simulation and optimization program for all scenarios based on AR devices. When the real-time energy consumption simulation and optimization program for all scenarios based on AR devices is executed by the processor, it implements the above-mentioned real-time energy consumption simulation and optimization method for all scenarios based on AR devices.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0017] 1. To address the issue of misclassification / device instance segmentation caused by similar equipment appearances or occlusion in existing industrial scenarios, this invention, after initial segmentation, constructs hierarchical associations and calculates consistency scores to accurately assess the reliability of the segmentation results. Combined with a secondary verification mechanism based on temporal information, it ensures the stability of energy consumption fluctuations and equipment operating states, thereby filtering potential errors. Then, through continuous incremental learning and parameter optimization, the model can dynamically adjust, significantly reducing the segmentation error rate. The final structured output not only includes high-confidence instance masks and category labels but also incorporates equipment operating parameters, enhancing the model's contextual relevance and stability in energy consumption scenarios. This provides high-precision, strongly correlated visual feature primitives for subsequent spatial alignment and energy consumption analysis, significantly improving the accuracy of equipment identification, reducing the need for manual verification, optimizing model performance, and ensuring the accuracy and reliability of energy consumption prediction.

[0018] 2. This invention uses the NTP protocol to precisely synchronize the clocks of all sensor devices, ensuring that all collected data have a unified time base. Simultaneously, it utilizes bilinear interpolation technology to map discrete data points from different sensors into a unified three-dimensional grid coordinate system, effectively integrating data from different sensors and ensuring spatial consistency. Secondly, in terms of anomaly suppression, a statistical threshold-based method is used to detect and process anomalies, reducing the impact of noise on the entire system and improving data quality and reliability. Finally, an extended Kalman filter algorithm is used for dynamic fusion of multi-source data. Utilizing a prediction and update loop mechanism, real-time estimation and updating of device status and environmental data are achieved, effectively handling uncertainties and noise in the data and further improving the accuracy of data fusion. Attached Figure Description

[0019] The disclosure of this invention is illustrated with reference to the accompanying drawings. It should be understood that the drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. In the drawings, the same reference numerals are used to refer to the same parts. Wherein:

[0020] Figure 1 This is a schematic diagram of the energy consumption simulation and optimization process proposed in one embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram illustrating the process of constructing an energy consumption device database including spatial location, visual entities, device IDs, and real-time data streams, as proposed in one embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram illustrating the process of constructing an incremental learning mechanism to improve the segmentation effect of a segmentation model, as proposed in one embodiment of the present invention. Detailed Implementation

[0023] It is readily understood that, based on the technical solution of this invention, those skilled in the art can propose various interchangeable structural methods and implementations without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention.

[0024] The present invention will be further described in detail below with reference to the accompanying drawings, but this is not intended to limit the scope of the invention.

[0025] like Figure 1 As shown, as an embodiment of the present invention, the first proposed method for real-time energy consumption simulation and optimization based on AR devices in all scenarios includes the following steps:

[0026] S1. Data Acquisition: Utilizing the multimodal sensor array of the AR device, including a high-resolution RGB camera and a depth sensor, visual data and depth data of the physical scene are acquired simultaneously. The acquired data is then standardized and preprocessed to ensure the accuracy and consistency of the data.

[0027] S2. Data Fusion: Fusion of preprocessed visual data and depth data to construct a 3D point cloud model representing the physical scene;

[0028] S3, Energy Consumption Device Database: Using a deep learning model, energy consumption category segmentation and device instance segmentation are performed on RGB images captured by high-resolution RGB cameras in AR devices. Energy consumption device elements in the physical scene are identified, and the segmentation results, 3D point cloud model data and device operation data are spatially aligned to build an energy consumption device database including spatial location, visual entity, device ID, and real-time data stream. This enables accurate identification of energy consumption devices in the scene, thereby providing detailed device information for energy consumption monitoring.

[0029] S4. Energy Consumption Prediction Simulation:

[0030] Based on the identified energy-consuming equipment information, a spatiotemporal sequence model is constructed to predict energy consumption. The Monte Carlo dropout method is used to generate confidence intervals, and the uncertainty in the energy consumption prediction process is quantified by combining point prediction values ​​with probability intervals.

[0031] S5. Continuously track the energy consumption data of various devices in the physical scene. Through the built low-latency data pipeline, continuously receive the operating parameter data of energy-consuming devices pushed by the multi-threaded IoT protocol interface, and perform real-time cleaning and preprocessing, and link it to the energy-consuming device database in real time.

[0032] S6. Based on the energy consumption prediction results, construct an optimization problem model with the objective function of minimizing energy consumption cost, total energy consumption, or peak demand, output the energy consumption equipment control strategy, and realize visualization based on the AR interface.

[0033] In one embodiment of the present invention, to provide precise positioning for optimizing the energy consumption of industrial equipment in a physical scene, data acquisition is achieved in step S1 through the following steps:

[0034] S1-1. Utilize the multimodal sensor array of the AR device to synchronously collect data within the physical scene, including:

[0035] Visual data: Visual data in the physical scene is captured using an industrial-grade RGB camera, and image frames I are acquired at multiple consecutive time points t=1,2,...,M. t To form a time-series image sequence .

[0036] Depth data: Spatial depth data characterizing the relative positions and distances between energy-consuming devices in a physical scene is acquired using ToF / structured light depth sensors. It should be noted that due to the optical characteristics of existing cameras, the captured images will exhibit varying degrees of distortion, mainly radial and tangential distortion, which in turn affects the overall energy consumption optimization effect.

[0037] Therefore, the visual data and depth data obtained in S1 need to be preprocessed to ensure the reliability of subsequent energy consumption analysis and optimization.

[0038] S1-2. Standardize and preprocess the collected data. In practice, the data preprocessing process follows a hierarchical calibration mechanism:

[0039] First, distortion correction is performed on the visual data: Based on the Zhang calibration model, at least 10 checkerboard pattern images (RGB images) from different angles are captured, and the corner coordinates are detected using the cornerDetect function of OpenCV. The least squares method is used to calculate the intrinsic parameter matrix K of the RGB camera, and radial-tangential distortion correction is performed on the images to reduce image edges and improve the accuracy of shape and position measurement of energy-consuming devices.

[0040] Secondly, spatial registration is performed by establishing a transformation relationship between the visual and depth coordinate systems to achieve precise registration between the depth data and the visual data, ensuring a high degree of consistency between them. Specific steps include:

[0041] a. Time-series image sequences acquired through synchronous acquisition Spatial depth data Using the focal length in the intrinsic parameter matrix K of the RGB camera and and the principal point coordinates of the image and The pixel coordinates representing the image location Depth values ​​representing the distance information from a pixel to the RGB camera Back projection into 3D point cloud:

[0042] At the same time, invalid depth points (such as...) are removed. ), generate a set of matching point pairs This facilitates subsequent conversion and registration between the visual and depth coordinate systems using these matching point pairs. For visual feature points, For the corresponding 3D point cloud, The coordinates along the z-axis in the depth coordinate system describe the position of the 3D point cloud in the depth direction (generally understood as the direction perpendicular to the imaging plane in space, pointing towards the depth of the energy-consuming device). The three coordinates of the 3D point cloud can accurately describe the specific orientation of the energy-consuming device in three-dimensional space. In practical implementation, the 3D point cloud can be regarded as the geometric representation of depth data in three-dimensional space. Depth data provides distance information, allowing the extension from the two-dimensional image plane to three-dimensional space, constructing a 3D point cloud that can describe the specific orientation of the energy-consuming device in three-dimensional space;

[0043] b. Solving the initial transformation parameters using OpenCV and It is understandable that the initial transformation parameters refer to the initial rotation and translation parameters used to describe the coordinate system transformation during the process of aligning and registering visual data with depth data. These parameters are obtained by acquiring the pose pairs of the RGB camera and depth sensor relative to the calibration board. r and t represent the rotation and translation parameters in three-dimensional space, respectively, used to describe the rotation angle and position movement of the camera during the coordinate system transformation. The purpose is to minimize the sum of squared Euclidean distances of all matching point pairs, achieve the initial alignment of visual data and depth data, thereby quickly establishing the initial transformation relationship between the two coordinate systems, reducing the overall offset of the 3D point cloud, providing reasonable initial values ​​for subsequent high-precision optimization, and avoiding iterative optimization from getting trapped in local optima.

[0044] c. Starting with the initial transformation parameters, using the IterativeClosestPoint(ICP) algorithm from the PCL library, the rotation parameters are continuously adjusted. The resulting rotation matrix R and translation parameters To minimize the objective function To optimize the objective, R and t are continuously adjusted to make the 3D point cloud... With visual feature points The distance error between them gradually decreases until the set convergence conditions are met, such as when the error threshold is less than a certain set value (e.g., 0.1 cm) or when the maximum number of iterations is reached (e.g., 50 times). At this point, the optimal transformation is solved to ensure the spatial consistency between the depth data and the visual data.

[0045] In one embodiment of the present invention, the construction of a three-dimensional point cloud model is achieved through the following steps:

[0046] S2-1, Feature Extraction and Tracking:

[0047] The ORB feature detector is used to extract FAST corner points in image frame It, and the set of visual feature points is obtained by traversing all pixels of the image. ,in Using pixel coordinates, and combining optical flow constraint equations for consecutive frames , Calculate feature points Cross-frame feature trajectory, and minimize grayscale error through iteration. Solve for the displacement vector of the feature point. To achieve cross-frame tracking, the average tracking length of the trajectory is set to be no less than 20 frames.

[0048] Meanwhile, at the image frame interval Internally, depth data is analyzed using the median integral method. Pre-integration optimizes the accuracy of depth data, reduces noise, minimizes error accumulation in the data, and yields the pose changes of energy-consuming devices in the physical scene, including rotational changes. speed change and position changes ;

[0049] S2-2. Construct an error function to minimize reprojection error and depth data pre-integration error, thereby optimizing the parameters of the energy-consuming device pose and 3D point cloud: In the formula, The state vector contains the camera-to-world coordinate system transformation matrix at all times and the coordinates of all 3D point clouds. For the Cauchy robust kernel function, 3D point cloud coordinates For visual reprojection error, it represents minimizing the 3D point cloud coordinates. After being projected by the camera, and the feature points distance, K is the intrinsic parameter matrix of the RGB camera. for Coordinates in the camera coordinate system These are the intrinsic parameters of the intrinsic parameter matrix after distortion correction in step S1. For reprojection error, The depth data residual term includes pre-integration errors of rotation, velocity, and position, which are solved iteratively using the LM algorithm. The transformation matrix from the camera to the world coordinate system at time t (including rotation) Translation This transformation matrix can convert acquired image data (such as visual feature points) and depth data (such as 3D point clouds) to a unified world coordinate system, enabling the fusion and alignment of multi-source data. and To optimize the speed and position, This represents the x-component of the camera's translation vector in the world coordinate system, that is, the amount of translation of the camera along the x-axis. This represents the z-component of the camera's translation vector in the world coordinate system, i.e., the amount of translation of the camera along the z-axis.

[0050] S2-3, Dense Point Cloud Generation: Utilizing the Optimized Pose With depth data For image pixels (U is the set of effective pixels) Inverse projection generates dense point clouds , Generate dense point clouds, Let K be the inverse matrix of the intrinsic parameter matrix. This represents the depth value at pixel u at time t. Represents the homogeneous coordinate form of pixel coordinate u. Representing pixel coordinates, typically a two-dimensional vector. .

[0051] S2-4, Point Cloud Filtering and Map Building: Setting Voxel Size For dense point clouds Downsampling is performed, retaining the geometric center or average point within each voxel. Statistical filtering is then used to calculate the average distance from each point to its 50 nearest neighbors. , Distance removal Mean +2 standard deviations point, Indicates dense point clouds The i-th point and the j-th point The vector difference between them is used to fuse dense point clouds from multiple frames. Construct point cloud density Dynamic global map M, M= , Represents point cloud data for all time points t. Perform a union operation on the sets to form a 3D point cloud model.

[0052] like Figure 2 As shown, in one embodiment of the present invention, the existing technology has the following defects in the energy consumption category segmentation of 3D models: single-modal segmentation models (such as those based solely on RGB images) do not integrate 3D geometric features, resulting in misalignment between semantic labels and spatial positions, and failure in occluded scene segmentation; at the same time, current multi-source data use offline post-processing fusion, which suffers from time synchronization loss and computational redundancy, failing to meet the real-time needs of industry and making it difficult to adapt to the addition of new equipment or changes in pose, resulting in the decoupling of semantic labels from point cloud positions, directly affecting the reliability of the association between the semantic information and spatial position of the 3D model, and restricting the accuracy of industrial energy consumption analysis. Therefore, the present invention further proposes step S3:

[0053] S3-1 employs a deep learning model to perform real-time energy consumption category segmentation on RGB images (visual data) collected by AR devices. This identifies and labels key elements in the scene, such as basic building elements and energy-consuming equipment elements, thereby associating equipment parameters with spatial location information and enhancing the model's interpretability. Energy consumption category segmentation is a method of segmenting visual data to identify regions in an image that fall into different energy consumption categories. For example, regions in an image can be divided into "high-energy-consuming equipment areas," "medium-energy-consuming equipment areas," and "low-energy-consuming equipment areas." This provides coarse-grained identification of devices with different energy consumption levels in the scene, helping to quickly locate and identify which areas contain key equipment affecting overall energy consumption. Device instance segmentation, building upon energy consumption category segmentation, further distinguishes different individual devices within the same energy consumption category. For example, in the "high-energy-consuming equipment area," each air conditioner and each CNC machine tool can be identified individually. This segmentation method provides fine-grained identification results, allowing each specific energy-consuming device to be located and tracked individually, providing a foundation for subsequent accurate energy consumption analysis and management.

[0054] Based on the above technical concept, in specific implementation, it is preferable to use a parallel dual-model approach of DeepLabv3+ network and MaskR-CNN to perform pixel-level category labeling and device instance segmentation on the RGB images acquired by the AR device, outputting instance masks and device instance segments to provide visual feature primitives for subsequent spatial alignment. This is achieved through the following steps:

[0055] Energy-intensive category segmentation aims to classify each pixel in an image, assigning it a specific category label, and segmenting the image into regions with different category meanings, thereby enabling the 3D point cloud model to understand the image content. In practice, a DeepLabv3+ network is used to classify the input image and output a category mask. ,in, Assigning scene element categories to corresponding building elements and energy-consuming equipment, such as labeling the point cloud of a CNC machine tool as "CNC machining equipment," an industrial robot as "automated assembly unit," and a material conveyor belt as "logistics transmission system," thereby achieving pixel-level category differentiation and recognition of equipment in three-dimensional space. Then, the mask is applied... Isolated noise points are removed by morphological opening operation (kernel size 3×3) while preserving the complete outline of scene elements;

[0056] Device instance segmentation aims to accurately separate each individual object in an image from the background and other objects, thus marking the boundaries and extent of each instance. In practice, the Mask R-CNN model is used to generate candidate bounding boxes to define the areas where energy-consuming devices are located in the physical scene. Then, fixed-size features are extracted from these candidate bounding boxes using the ROIAlign layer, and classification (e.g., device type, such as air conditioner, lighting, etc.) and mask prediction are performed in parallel to obtain the instance mask. , Let H be the feature map (instance) of the i-th feature variable, W be the feature map height, W be the feature map width, and N be the number of feature map channels / number of instances. For the instance mask... The i-th instance has a corresponding binary mask channel of ∈{0,1} H×W This is to accurately identify the pixel position occupied by the instance;

[0057] S3-2. Use step S1 to correct the intrinsic parameter matrix K and the optimized pose in step S2. With depth data For the obtained category mask (Category mask assigns a specific category label to each pixel (such as "high-power device area", "medium-power device area", and "low-power device area") and instance mask) (Instance mask marks the device instance to which each pixel in the image belongs, such as air conditioner, CNC machine tool) Covering each valid pixel Combined with its corresponding depth value Through three-dimensional spatial geometric transformation, it is converted into 3D coordinates in the camera coordinate system, and then the transformation matrix from camera to world coordinate system is optimized through step S2. Converting these points to world coordinates allows for the mapping of category and instance information to a 3D point cloud, resulting in a 3D point cloud set for that instance. , , Let K be the inverse matrix of the intrinsic parameter matrix. The pixel coordinate u is represented in homogeneous coordinate form. Then, the centroid of the point cloud is calculated using a weighted average. , serving as the spatial coordinates of energy-consuming equipment, where, Here, m represents the 3D point cloud coordinates in the world coordinate system, m is the number of points in the instance 3D point cloud, and m represents the mask. The total number of pixels in the image.

[0058] Example: Continuing with the example of an automobile manufacturing workshop, based on the concepts in S3-1 and S3-2 above, firstly, energy consumption category segmentation is performed on the RGB image of the workshop to identify the pixel regions corresponding to different categories of equipment, such as air conditioners and lighting. However, the category mask can only provide pixel-level category information and cannot distinguish different instances within the same category. For example, if there are two CNC machine tools, the category mask can only identify the category "CNC machine tool," but cannot distinguish between "CNC machine tool 1" and "CNC machine tool 2." This is insufficient for energy consumption prediction and management because different instances have different energy consumption characteristics and operating states. Therefore, based on energy consumption category segmentation, further distinctions are made between different individual devices within the same energy consumption category. For example, in the "high-energy-consuming equipment area," each specific CNC machine tool is identified separately and marked as a different instance (e.g., "Instance 1," "Instance 2") to provide fine-grained identification results, rather than lumping all devices of the same category together. Next, the centroid coordinates of each instance mask are calculated. Assuming that the air conditioner mask of Instance 1 has multiple pixels, the average of the coordinates of these pixels is the centroid coordinates of that instance. Then, depth data fed back from the depth sensor and camera pose obtained through optimization algorithms are acquired. Using the corrected intrinsic parameter matrix and this data, the effective pixels covered by the category mask and instance mask are determined. The corresponding depth values ​​are converted into 3D coordinates in the camera coordinate system. Finally, using an optimized transformation matrix, the 3D points in the camera coordinate system are transformed into the world coordinate system, resulting in the final 3D point cloud. At this point, these 3D point clouds achieve the mapping of category and instance information in space.

[0059] S3-3, Based on the obtained point cloud and center of mass The process of creating database entries for energy-consuming devices is as follows:

[0060] First, the point cloud set of the device Iterate through all points and find the minimum and maximum coordinate values ​​of each point in the x, y, and z dimensions. These six values ​​together constitute a spatial boundary box describing the spatial range of energy-consuming devices, thereby defining the location and range of instances in space.

[0061] Then, the instance's category label, such as "CNC machine tool", unique ID, such as "AirCondition_001", centroid coordinates and spatial bounding box parameters are stored in the node to generate a node attribute structure as shown in Table 1 below, thereby constructing an energy consumption simulation system and creating detailed database entries for each device. Table 1:

[0062] It is understandable that the 3D point cloud set based on step S3-2 A graph database containing spatial topology and data flow associations is constructed, combining the device category labels, instance types and their corresponding 3D point clouds in the world coordinate system to create detailed database entries for each device. At this point, the spatial location, category mask and instance information of the device are integrated and stored, thereby providing a structured data foundation for subsequent access to more device-related data and energy consumption analysis.

[0063] In one embodiment of the present invention, a multi-threaded IoT protocol interface is used to collect various types of device operation data in a physical scene in parallel using the multimodal sensor array of the AR device. Device operation data (electrical parameters, temperature parameters, flow parameters, and device status parameters) are collected at different frequencies and in different ways to ensure real-time data transmission. The device operation data is then fused with a 3D point cloud model. Through timestamps and spatial location information, various parameter data are associated with corresponding positions in the 3D point cloud model, forming a full-scene digital twin with point cloud as its framework and multi-source parameters as its attributes. This provides a unified spatiotemporal reference for energy consumption analysis.

[0064] In practice, it is preferable to use an extended Kalman filter framework to fuse the multi-source data mentioned above, ensuring data consistency and accuracy. The specific steps are as follows:

[0065] a. Perform sensor data preprocessing: First, perform spatiotemporal registration on the data collected by the multimodal sensor array (such as light, temperature, and humidity sensors), and synchronize the clocks of all sensors using the NTP protocol to ensure consistency of the time base; second, since the sensor sampling frequencies and coordinate distributions vary, bilinear interpolation is also needed to map the discrete sensor measurements to a unified three-dimensional grid coordinate system: for the sensor data set The measured value of each sensor i Based on its corresponding spatiotemporal coordinates ,in For the physical location of the sensor, For the sampling time, a bilinear interpolation algorithm is used to map discrete measurements to grid nodes; then, synchronous data is filtered using a time window filter (window length ∆T), taking into account the sensor's maximum sampling period and data transmission delay for the sensor data set. Iterate through all data records, taking the sampling time of each data record as an example. Based on the time interval, extract Data from other sensors is incorporated into a temporary dataset, and the dimensions are constructed as follows: The spatiotemporal correlation matrix (where n is the number of sensors and m is the number of data records after time synchronization) is defined, and the matrix elements represent the measurement value of the i-th sensor at the j-th synchronization time point. When there is no valid data, NaN is filled in, and outliers are detected and repaired based on a statistical threshold method.

[0066] b. Achieve effective fusion of multi-source data: First, taking the data collected by the above-mentioned light, temperature, and humidity sensors as an example, construct a point cloud centroid containing light intensity L, temperature T, humidity H, and the acquired device data. The state vector X, Secondly, the prediction step is executed. Using the state transition matrix F, and based on the temporal correlation of environmental parameters (such as temperature gradient changes), the formula is used... Derive the prior estimate of the state. In the formula, Given the posterior state of the previous time step, derive the prior estimate of the state; then, perform an update step, using the light intensity values ​​measured by the sensor. ,temperature ,humidity and centroid measurement value observation vector Through Kalman gain In the formula, Let H be the prior covariance, H be the observation matrix, and R be the observation noise covariance. Combined with the formula... The state estimates are corrected to achieve effective fusion of multi-source data (environmental data and centroid spatial data), generating a structured dataset that includes spatial, equipment, environmental, and temporal data. This allows for a more comprehensive reflection of the physical scene information. At this point, based on the energy consumption simulation system, as shown in Table 2 below, the database entries for devices within the physical scene have been created. Table 2:

[0067]

[0068] It should be noted that the state transition matrix F needs to be constructed based on environmental parameters and the physical laws of the change of the centroid over time (such as the evolution model of temperature over time, the movement law of the equipment centroid due to production activities, etc.). The observation matrix H is used to describe the mapping relationship from the state vector to the observation vector. If each parameter can be directly measured and corresponds, then H can be set as an identity matrix.

[0069] Based on the above technical concept, it should be noted that in step S3-1, after energy consumption category segmentation and device instance segmentation, preliminary category masks and instance masks are obtained. However, this preliminary segmentation result has the following problems:

[0070] In the energy consumption category segmentation results, there are noise points and blurred boundaries. For example, in complex industrial scenarios, similar-looking equipment (such as CNC machine tools and ordinary machine tools) may be incorrectly classified in some areas due to similar shapes and colors, which will reduce the accuracy of energy consumption category segmentation. Secondly, in the equipment instance segmentation results, there are also cases of inaccurate contours, missed detections, or false detections. Small equipment arranged closely together may be mistakenly merged into one instance, while large equipment may be incorrectly segmented into multiple instances due to occlusion or deformation, thereby reducing the accuracy of equipment instance segmentation.

[0071] The direct result is that the initial segmentation incorrectly associates instances with the wrong category mask (e.g., misclassifying a "CNC machine tool" instance as a "general machine tool"), or omits the association between instances and the category mask (e.g., failing to identify some equipment instances belonging to the "energy-consuming equipment" category), weakening the relevance of the segmentation results. Finally, if these initial segmentation results with biases, noise, and inaccurate associations are directly used for subsequent spatial alignment, it will lead to inaccurate spatial alignment, thereby affecting the association of equipment parameters and the performance of subsequent functional modules, ultimately reducing the accuracy of the system's analysis and processing of information such as equipment energy consumption and operating status.

[0072] like Figure 2 As shown, based on this, the initial category mask is output. and instance mask Furthermore, it is necessary to process the following segmentation optimization mechanism to continuously evolve the deep learning model, reduce the subsequent segmentation error rate, and thus provide high-precision, strongly correlated visual feature primitives for subsequent spatial alignment.

[0073] The process is as follows:

[0074] S3-11, Output category mask As a global feature representation, it is used to represent the image. Overall, coarse-grained category structure information, expressed as an output instance mask. Local feature representation, used to represent images Specific energy-consuming devices and fine-grained instance information are used to construct hierarchical relationships: the fine-grained category label (class) of each instance i is... i Mapped to a category of a predefined overall structure In Chinese, for example, the class "CNC machine tool" i Mapped to "energy-consuming device" c, denoted as: c = M(class i ), where M is a predefined mapping function.

[0075] S3-12, For each instance i detected by the MaskR-CNN model (including the instance mask) Category tags i , Mapping class mask ), calculate its relationship with the class mask Consistency score in region C Region C is the category mask. The pixel region belonging to class c is considered as the set of all pixels predicted to be class c in the class mask. In the following formula, this region C is represented by a binary mask image. This indicates that the consistency score measures the agreement between the predicted category and the expected category c within an instance region. The calculation process includes:

[0076] First, let's start with the category mask. Generate category mask binary mask : ,

[0077] Second, calculate the instance mask. With category mask binary mask The intersection-union ratio is used to obtain the consistency score. This measures the degree of overlap between the segmented region of a device instance and the corresponding category mask region. In the formula, For logical AND operation, it represents the category mask. and The intersection of the two masks represents the overlapping region C containing the instance mask and the class mask. The logical OR operation represents the category mask. and binary mask The union of the instance mask and the class mask represents the total coverage area of ​​region C. This indicates the segmented region of the device instance and the corresponding category mask region. The degree of overlap of coverage, when When =1, it represents an instance mask. Complete with category mask region overlap, when When it is = 0, it represents the instance mask Completely non - overlapping with the class mask region Completely non - overlapping

[0078] Example: Assume the instance mask And the class mask region Are two binary masks, as follows:

[0079] With a size of 3×3, where the pixel positions with value 1 are (1,1), (1,2), (1,3), (2,1), (2,2), (3,1), With a size of 3X3, where the pixel positions with value 1 are (1,1), (1,2), (2,1), (2,2), (3,1), (3,2), then, we get The overlapping pixel positions are (1,1), (1,2), (2,1), (2,2), (3,1), and the number of pixels in the determined intersection is 5. Similarly, we get The covered pixel positions are (1,1), (1,2), (1,3), (2,1), (2,2), (3,1), (3,2), and the number of pixels in the determined union is 7. The final obtained consistency score = 5 / 7 ≈ 0.714, and we get the example mask And the class mask region The overlapping degree is 71.4%.

[0080] S3 - 13. Set the consistency threshold Tcon, such as Tcon = 0.8. If the calculated , then it is considered that the segmentation result of this instance is reliable, and this instance is retained;

[0081] On the contrary, it is determined that there are potential errors or ambiguities in the segmentation result of instance i. For example, the instance is segmented into the wrong class region, the class prediction is incorrect, or the mask is inaccurate. At this time, based on the consistency score , through the formula , assign an initial confidence weight W to each instance i i , where k is a scaling factor used to control the steepness of the function, and W i ∈(0,1). Generally speaking, when S i ≥ Tcon, take W i ≈ 1, indicating that the confidence of the instance is relatively high and it is included in the candidate correct instance set. And trigger the following process:

[0082] Secondary verification: For instances with S i < Tcon, use temporal information, consecutive frames Perform secondary verification on the instance trajectory stability in []. If all the low-confidence instance information to be verified in the current batch has been verified (the scores are all higher than the threshold), mark them as reliable instances, add them to the candidate correct instance set, and terminate the optimization mechanism. Otherwise, mark the unverified instance information as the Uncertain state for manual review and processing. After completing the processing of the instances in the current batch, the optimization mechanism terminates.

[0083] Based on the above technical concept, it can be understood that W i The purpose is to quantify the preliminary reliability assessment based only on the visual consistency of the current frame: W i Close to 1 indicates less visual ambiguity, W i Close to 0 indicates extremely poor visual consistency, providing priorities or initial states for subsequent processing (especially secondary verification). It should be noted that in practical applications, to optimize computing power resources, the quantification of Si "much less than" Tcon can be achieved by setting a significant deviation threshold. Take W i ≈0, indicating that the confidence of the instance is low and it is directly included in the missegmented instance set, W i Instances close to 0 are directly regarded as errors and do not require verification or have low-priority verification; while W i Instances close to Tcon are more worthy of verification. Subsequently, for these S i <Tcon instances, trigger a secondary verification process based on the temporal energy consumption stability (analyze the power mean, variance, and correlation ρ with time in consecutive frames, etc.). If the instance passes the secondary verification (the energy consumption is stable and meets the preset criteria), update its confidence weight (formula 10 below), and mark the instance as reliable and add it to the candidate correct instance set; if the verification fails or the energy consumption is abnormal, mark the instance as the Uncertain state and wait for manual review.

[0084] So far, the finally output structured segmentation result combines the precise boundary masks of high-confidence instances instance masks and the fine-grained category labels class i , and also annotates the category c to which each instance belongs and its confidence weight , and retains the overall energy consumption category segmentation result , thereby enhancing the context relevance of the segmentation result and providing high-precision and strongly category-associated visual feature primitives for subsequent spatial alignment and device parameter association.

[0085] In an embodiment of the present invention, the process of performing secondary verification using temporal information is as follows:

[0086] First, traverse the consecutive frame sequence for a time window such as (corresponding to 10 frames, frame rate 5fps) Collect time-series energy consumption data from the equipment. ∈[t−Δt,t+Δt], based on sensor array acquisition device in The real-time power at time is This yields the energy consumption time-series data of energy-consuming devices within a continuous time window.

[0087] Secondly, calculate the average energy consumption (power) of energy-consuming devices within the time window. and variance To measure the stability and fluctuation of equipment energy consumption: , In the formula, For time series variables, it represents traversing each time point within a time window;

[0088] Next, calculate the correlation coefficient between the energy consumption time series and the time series. This is to verify the stability of device energy consumption over time, thereby determining the reliability of the device instance segmentation results. In the formula, The value ranges from [0,1] and measures the linear correlation between energy consumption and time. Let t be the mean of the time series, and t be the current time point.

[0089] Therefore, a consistency judgment is made: through quantitative analysis. Mean correlation coefficient with similar historical examples , correlation coefficient standard deviation of similar historical examples Jointly determine the relationship:

[0090] When satisfied (k is an adjustable hyperparameter; a value of k=1.5 is recommended, covering approximately 87% of normally distributed data), and , To preset an energy consumption fluctuation threshold for determining whether energy consumption changes are abnormal, an instance confidence weight update operation is performed according to the following formula. At this point, the device energy consumption is considered stable, indicating that the device instance segmentation result is reliable. Verification shows that the device instance segmentation result has stability and accuracy in continuous video streams. Simultaneously, instances meeting the above conditions are included in the candidate correct instance set. Store data in the form of triples: entity, relation, category, such as CNC machine tool, belonging to energy-consuming equipment. In the formula, For the equipment in Real-time power at any given moment For the equipment in Real-time power at any given moment The updated confidence weight for instance i.

[0091] If the correlation coefficient Exceeding the historical data distribution range ( Or abnormal energy consumption fluctuations may occur. If the instance segmentation result is incorrect, the instance is marked as "Uncertain" and a manual review process is triggered. Instances confirmed by manual review are then added to the erroneous segmentation instance set. In the form of quintuples , Describe the error type, such as a safety identifier being incorrectly mapped to an energy-consuming device: =(Security Identification Mask, Energy Consumption Device Area, Security Identification, Non-Energy Consumption Device, Category Mismapping), or lighting equipment is incorrectly mapped as a high-energy consumption device. =(Lighting equipment mask, high energy consumption equipment area, lighting equipment, medium energy consumption equipment, misclassification of energy consumption level), store it, and if the number of instances that need to be manually reviewed every day drops to less than 20% of the initial amount, then stop model optimization or reduce the optimization frequency.

[0092] like Figure 3 As shown, in one embodiment of the present invention, to further improve the accuracy and reliability of the model in energy consumption prediction, a daily calibration mechanism based on incremental learning is also needed. This mechanism injects the knowledge of the aforementioned sets of correct and incorrect instances into the loss function, dynamically adjusting the model parameters. Specifically, a repulsive penalty is applied to incorrect instances, while an attractive penalty is applied to correct instances, thereby optimizing the model parameters until the model's error rate significantly decreases, thus improving the accuracy of subsequent energy consumption predictions. The process is as follows:

[0093] First, from the correct set of instances Extract all correct instances stored in triplet form (e.g., ⟨CNC machine tool, belonging to, energy-consuming equipment>), and generate entity feature embedding vectors using the ResNet-50 backbone network of Mask R-CNN. Simultaneously, from the incorrectly segmented instance set Obtain incorrect segmentation cases in quintuple format (e.g., ( , (Safety identifiers, non-energy-consuming devices, category mismapping) are corrected by generating target embedding vectors using the ASPP module in DeepLabv3+. Fusion equipment operating parameters (real-time power) Correlation coefficient To construct the daily incremental training dataset ;

[0094] Secondly, through the multimodal loss function framework Continuous optimization of model updates, including

[0095] Basic segmentation loss function Used to measure the model's performance in pixel classification tasks, the cross-entropy loss function optimizes the model's classification accuracy for each pixel: In the formula, N is the total number of pixels, that is, the number of pixel samples involved in the calculation. This is the true label of the i-th pixel (1 if it belongs to the target category, 0 otherwise), used to indicate the actual classification of that pixel. The model predicts the probability of the i-th pixel, representing the confidence level of the model in judging that the pixel belongs to the target category. When the model prediction differs greatly from the true label, the loss value increases significantly.

[0096] Error suppression loss function This is used to correct prediction bias in the model on erroneous instances by penalizing the distance between erroneous feature embeddings, reducing repeated outputs of erroneous patterns by the model. For example, it forces the model to move the "safety identifier" entity closer to the "non-energy-consuming device" category in the feature space, thus eliminating mismapping. ;

[0097] Correctly enhance the loss function This is used to enhance the model's ability to identify correct instances by maximizing the similarity between correct feature embedding pairs, thereby increasing the model's confidence in correct patterns. For example, it can enhance the embedding correlation between "CNC machine tool" and "energy-consuming equipment". , It is the sigmoid function;

[0098] Next, gradient backpropagation is performed to optimize the weights of the ROIAlign layer in Mask R-CNN (to enhance sensitivity to error regions) and the parameters of the ASPP module in DeepLabv3+ (to strengthen the association with correct classes), and an adaptive learning rate is adopted. Execution parameter update , The gradient of the total loss function with respect to the parameters is used to immediately validate the mIoU with unlabeled samples after the update (if the decrease exceeds 5%, roll back and restore the parameters of the previous version) and continue optimization. Otherwise, it indicates that the model performance has stabilized and optimization is stopped to ensure the stability of model updates.

[0099] Finally, monitor the error recurrence rate. When the error recurrence rate When the recurrence rate is below 10% of the threshold, the preset energy consumption fluctuation threshold is increased to strengthen the screening criteria, and model optimization is stopped. When the error recurrence rate exceeds 10%, the preset energy consumption fluctuation threshold is decreased to relax the screening criteria, and optimization continues. This is the set of error instances detected in the new cycle. This is the set of historical error instances, representing the set of error instances recorded by the model within a historical period.

[0100] In one embodiment of the present invention, step S4 is further proposed to construct and train a ConvLSTM-GNN hybrid model to achieve accurate prediction of energy consumption data. The specific implementation steps include:

[0101] S4-1, For structured data groups Multi-dimensional feature extraction and analysis are performed to extract key feature vectors that accurately describe energy consumption scenarios, thereby obtaining detailed information on equipment operating status, environmental data, time periodicity, and historical energy consumption trends. The process is as follows: First, time feature processing: timestamps are converted into periodic codes, and trigonometric functions are used to cyclically encode hours, week numbers, and months, incorporating periodic patterns. Simultaneously, Boolean values ​​for holidays are overlaid to distinguish special dates, integrating these into a time feature set. Second, an equipment operating status vector is constructed. To comprehensively characterize equipment operating conditions: One-hot encoding is used to characterize equipment operating modes, denoted as... If an energy-consuming device is in different operating modes such as cooling, heating, or standby, set the corresponding dimension of its encoding vector to 1, and the other dimensions to 0; for example, set the device load rate... Start-stop status (Value 1 indicates running status, 0 indicates stopped status) and temperature setpoint As a numerical feature, it is related to the operating mode encoding vector. Together they form the state vector ,Right now Secondly, standardize the environmental characteristic matrix: such as organizing indoor temperature. ,humidity ,illumination and outdoor temperature ,humidity Wind speed Precipitation probability Forming an environmental feature matrix .

[0102] Z-score standardization is used to eliminate the influence of dimensions. The formula is: In the formula, For the standardized i-th environmental feature, This is the historical average of this feature. The historical standard deviation is used; finally, time, equipment status, and environmental data are integrated to generate key feature vectors. .

[0103] S4-2, ConvLSTM-GNN Model Construction and Training:

[0104] First, ConvLSTM temporal encoding is performed using 32 3×3 convolutional kernels. The cell state and hidden state are dynamically updated with the help of a gating mechanism to achieve the extraction of temporally dependent features.

[0105] Secondly, a fully connected output and training configuration is performed to construct a 2-layer 64-dimensional GNN, and the features of the GNN output nodes are integrated and mapped to energy consumption prediction values ​​through a fully connected layer. Meanwhile, the Huber loss function is used to balance the prediction error, as shown in the formula: In the formula, The threshold value represents the balance between mean squared error and mean absolute error characteristics. When configuring, the optimizer chosen is AdamW, and the learning rate is... Weight decay To suppress overfitting, the dataset is divided into 70% (training), 15% (validation), and 15% (test). The maximum training cycle is set to 200 rounds, and the batch size is 256. Training is stopped if the validation set loss does not decrease, in order to ensure the model's generalization ability.

[0106] S4-3. Employing a tiered strategy to achieve accurate prediction of equipment energy consumption:

[0107] For short-term predictions (e.g., 1–4 hours, 5-minute granularity), and to meet high-frequency real-time scheduling requirements, a 5-minute granularity feature vector is used. As input, the ConvLSTM-GNN model is used to capture fine-grained temporal fluctuations, and the output is a 48-step predicted sequence. This enables real-time demand response;

[0108] For medium- to long-term forecasts (e.g., 1-7 days, hourly granularity), the focus is on equipment maintenance and energy efficiency assessment, generating hourly features by downsampling the original features. By using a 4-layer encoder-decoder and an 8-head self-attention mechanism to mine its long sequence dependencies, it outputs a 168-step prediction sequence, thereby achieving real-time demand response.

[0109] Example: Taking a large air conditioning system in automobile production as an example, this system adjusts its energy consumption every 5 minutes based on changes in the indoor and outdoor environment and personnel activity. To achieve refined energy management, it is necessary to make short-term predictions of energy consumption for the next 4 hours in order to adjust the operating strategy in a timely manner and meet real-time demand response. In this physical scenario, the feature granularity is set to 5 minutes. The energy consumption data every 5 minutes and related influencing factors (such as historical energy consumption, indoor and outdoor temperature, humidity, and personnel flow in the same time period) constitute the feature vector xt. A ConvLSTM-GNN model is used. During implementation, the 5-minute granularity feature vector xt is input into the model to capture fine-grained temporal fluctuations and output the prediction sequence for the next 48 5-minute time steps (i.e., 4 hours). This allows managers to know in advance the energy consumption for each 5-minute period within the next 4 hours, making it easier to adjust the operating parameters of the air conditioning system in a timely manner to cope with changes in real-time demand and achieve energy conservation.

[0110] Based on the above technical concepts, traditional models in equipment energy consumption prediction typically only provide a single-point prediction result, lacking quantification of prediction uncertainty. For example, a simple linear regression model might predict that a device will consume 100 kWh of electricity in the next hour, but it doesn't indicate the reliability of this prediction or whether there is significant uncertainty. The direct problem is that a single prediction result has obvious shortcomings in practical applications: when the prediction deviates significantly from the actual situation, it leads to decision-making errors. For instance, in energy management, failure to accurately predict the fluctuation range of energy consumption can result in insufficient or excessive energy supply, thus affecting the stable operation of the system and the rational utilization of resources.

[0111] Based on this, this invention further proposes S4-4, employing the Monte Carlo dropout method to generate confidence intervals, quantifying the uncertainty in the energy consumption prediction process by combining point predicted values ​​with probability intervals. The process is as follows:

[0112] First, construct a neural network module with a Dropout layer, and during the fully connected output and training configuration of the above model, set the Dropout probability, and use historical energy consumption data to train and optimize the model;

[0113] Secondly, during the prediction phase, the Dropout probability remains constant, and multiple (e.g., 100) random forward propagations are performed on the same sample. (Including historical energy consumption data, temperature, humidity, and other features of the device), multiple forward propagations are performed. Because the Dropout layer randomly discards different neurons during each propagation, the output prediction value varies. Since there are differences among (i=1,2,…,100), the statistic and confidence interval are calculated using the following formula: Point Predicted Value Next, the 100 predicted values ​​are sorted, and quantiles are calculated using linear interpolation to obtain confidence intervals at different confidence levels, such as the 95% confidence interval. and 80% confidence interval ;

[0114] Finally, the predicted point values ​​under this confidence interval As the main result of energy consumption forecasting, it is used for daily management, and the uncertainty of the forecast is quantified based on the confidence interval, so as to provide the credibility of the forecast results for energy consumption decision-making.

[0115] In one embodiment of the present invention, in order to achieve accurate visualization and real-time monitoring of energy consumption data, the system also needs to build a low-latency data pipeline (such as using the stream processing framework Apache Kafka for real-time data processing and analysis, configuring 3 partitions and 3 replicas, and setting a throughput of 10,000 messages / second), receive energy consumption and device status data pushed by the IoT gateway, and perform real-time cleaning and preprocessing. Among them, the Z-score method is preferred for outlier detection, missing values ​​are filled in according to type, and the calculation latency is controlled within 20ms. Finally, the cleaned data is associated with a database based on binding relationships, and an optimization problem model is constructed with minimizing energy consumption cost, total energy consumption, or peak demand as the objective function. The energy consumption device control strategy is output and visualized based on an AR interface.

[0116] In one embodiment of the present invention, as a second aspect of the present invention, a real-time energy consumption simulation and optimization system for all scenarios based on AR devices is proposed, including a memory and a processor, wherein the memory includes a real-time energy consumption simulation and optimization program for all scenarios based on AR devices, and when the real-time energy consumption simulation and optimization program for all scenarios based on AR devices is executed by the processor, the real-time energy consumption simulation and optimization method for all scenarios based on AR devices described above is implemented.

[0117] The technical scope of this invention is not limited to the content described above. Those skilled in the art can make various modifications and variations to the above embodiments without departing from the technical concept of this invention, and all such modifications and variations should fall within the protection scope of this invention.

Claims

1. A method for real-time energy consumption simulation and optimization across all scenarios based on AR devices, characterized in that: Including the following steps: By utilizing the multimodal sensor array of AR devices, visual data, depth data, and operational data representing target energy-consuming devices in physical scenes are collected to construct a three-dimensional point cloud model representing the scene where the energy-consuming devices are located. The collected visual data is segmented by energy consumption category and device instance. A segmentation optimization mechanism is constructed to improve the accuracy of the segmentation results and identify energy-consuming device elements in the physical scene. The execution process of the segmentation optimization mechanism is as follows: The overlap ratio between the instance mask output by device instance segmentation and the corresponding category binary mask output by energy consumption category segmentation is used to measure the degree of coverage overlap between the region where the device instance segmentation is located and the region where the corresponding category mask is located. A consistency threshold is set. If the degree of coverage overlap is higher than the threshold, the instance information is retained and output as the structured segmentation result. The process of building an energy-consuming device database to accurately identify energy-consuming devices in a scenario is as follows: After spatial alignment of the optimized segmentation results, 3D point cloud model data, and operational data, they are converted into world coordinate system points to map category and instance information to the 3D point cloud, resulting in a 3D point cloud set of instance information. Next, the centroid of the 3D point cloud is calculated using a weighted average and used as the spatial coordinates of the target device. Finally, based on the obtained point cloud and centroid, energy consumption device database entries are created: the 3D point cloud set is traversed, and the minimum and maximum coordinate values ​​of each point cloud in the x, y, and z dimensions are found to form a spatial bounding box, defining the location and range of instance information in space. The category label, unique ID, centroid coordinates, and spatial bounding box parameters of the instance information are stored in the node, constructing the device database entries of the energy consumption simulation system, achieving strong binding between the segmentation results, 3D point cloud model data, and device operational data. Based on the identified energy-consuming equipment information, a spatiotemporal sequence model is constructed to predict energy consumption, and the energy consumption prediction results are visualized based on an AR interface. Calculate the instance mask M i With category mask M s After the intersection and union comparison, the consistency score S is obtained. i This allows for the measurement of the degree of coverage overlap. If the degree of coverage overlap is lower than the consistency threshold Tcon, the segmentation result of instance i is determined to have a potential error or ambiguity. In this case, a consistency score S is required. i Configure confidence weights W for each instance i. i And trigger the following process: For instance information with potential errors or ambiguities, continuous frame analysis is performed using the collected temporal information containing visual data to perform secondary verification of the instance information trajectory stability in the continuous frames. If all instance information is verified and the scores are all higher than the threshold, the optimization mechanism is terminated; otherwise, unverified instance information is marked as "Uncertain" for manual review. The secondary verification process is as follows: First, energy consumption devices based on sensor arrays are... The real-time power at time is First, obtain the time-series energy consumption data of the energy-consuming equipment within a continuous time window; second, calculate the average energy consumption of the energy-consuming equipment within the time window. and variance First, it measures the stability and fluctuation of equipment energy consumption; second, it calculates the correlation coefficient between energy consumption time series and time series. The reliability of the device instance information segmentation results is determined; finally, quantitative analysis is performed. Mean of correlation coefficient with similar historical instances Standard deviation Consistency determination is made regarding the relationship.

2. The method for real-time energy consumption simulation and optimization based on AR devices in all scenarios according to claim 1, characterized in that: The correlation coefficient The calculation formula is: In the formula, The mean of the time series. Let t be the radius of the time window, and t be the current time point.

3. The method for real-time energy consumption simulation and optimization based on AR devices in all scenarios according to claim 1, characterized in that: The execution process for consistency determination is as follows: When satisfied ,and , To preset the energy consumption fluctuation threshold, an instance confidence weight update operation is performed. At this point, the device instance information segmentation result is considered reliable and passes verification. Simultaneously, instance information that meets the above conditions is included in the candidate correct instance set. , and store, where k is an adjustable hyperparameter; like or If the device instance information segmentation result is incorrect, the instance is marked as "Uncertain" and a manual review process is triggered. Instances confirmed by manual review are then included in the set of incorrectly segmented instances. , and store.

4. The method for real-time energy consumption simulation and optimization based on AR devices in all scenarios according to claim 3, characterized in that: If instance information is not verified and the score is lower than the preset threshold, or if the number of instances confirmed by manual review to have errors reaches the maximum set number, it is necessary to improve the segmentation effect of the segmentation model based on the constructed incremental learning mechanism. The process is as follows: First, based on the correct set of instances and incorrect segmentation instance set Constructing the daily incremental training dataset Secondly, through the framework of multimodal loss function. The segmentation model is continuously optimized, including the basic segmentation loss function. In the formula, N is the total number of pixels. Let i be the true label of the i-th pixel. Let be the model's predicted probability for the i-th pixel. When the model's prediction differs greatly from the true label, the loss value increases significantly. The error suppression loss function; To correctly enhance the loss function, gradient backpropagation is performed again to optimize the weights of the ROIAlign layer and the parameters of the ASPP module in the segmentation model. An adaptive learning rate is used to update the parameters, and mIoU is immediately validated with unlabeled samples after each update. Optimization continues until the model performance stabilizes, at which point optimization is stopped to ensure the stability of model updates. Finally, the error recurrence rate is monitored. When the error recurrence rate When the recurrence rate is below the recurrence rate threshold, the preset energy consumption fluctuation threshold is increased to strengthen the screening criteria, and model optimization is stopped. When the error recurrence rate exceeds the recurrence rate threshold, the preset energy consumption fluctuation threshold is reduced to relax the screening criteria, and optimization continues.

5. The method for real-time energy consumption simulation and optimization based on AR devices in all scenarios according to claim 1, characterized in that: In the process of building an energy consumption equipment database, it is necessary to achieve data fusion through an extended Kalman filter framework to ensure data consistency and accuracy. The process is as follows: First, sensor data preprocessing is performed, including spatiotemporal registration of data collected by the multimodal sensor array, and synchronization of all sensor clocks using the NTP protocol to ensure time reference consistency. Second, discrete sensor measurements are mapped to a unified 3D grid coordinate system using bilinear interpolation, and synchronized data is filtered using time window filtering to construct a spatiotemporal correlation matrix, detecting and correcting outliers in the sensor data. Third, a centroid containing energy consumption equipment operation data and 3D point cloud data is constructed. The state vector is used to perform a prediction step, which derives the prior estimate of the state using the state transition matrix. Then, an update step is performed to correct the state estimate by combining the Kalman gain with the observation vector, thus achieving effective fusion of multi-source data. Finally, a structured data set is generated to complete the creation of database entries for devices in the physical scene.

6. The method for real-time energy consumption simulation and optimization based on AR devices in all scenarios according to claim 1 or 5, characterized in that: The process of constructing a spatiotemporal sequence model for energy consumption prediction is as follows: First, feature extraction is performed on the structured data set to obtain key feature vectors representing energy-consuming devices in the physical scene. Second, using the ConvLSTM-GNN model, the key feature vectors are temporally encoded using ConvLSTM to extract temporally dependent features representing the temporal correlation and dynamic change patterns of the key feature vectors. These features are then integrated with the GNN output node features through a fully connected layer and mapped to predicted energy consumption values. Third, the Huber loss function is used to balance the prediction error, and the AdamW optimizer is used for training to ensure the model's generalization ability. Finally, a hierarchical strategy is constructed to achieve accurate prediction of device energy consumption.

7. The method for real-time energy consumption simulation and optimization based on AR devices in all scenarios according to claim 6, characterized in that: In the process of energy consumption prediction using the aforementioned spatiotemporal sequence model, the Monte Carlo dropout method is also required to generate confidence intervals. These intervals are then combined with point prediction values ​​to quantify the uncertainty in the energy consumption prediction process. The process is as follows: During the construction of the spatiotemporal sequence model for fully connected output and training configuration, the Dropout probability is set, and historical energy consumption data is used to train and optimize the spatiotemporal sequence model; during the prediction phase, the Dropout probability is kept constant, and the same sample is... The second random forward propagation calculates the predicted point value using a formula. Next, Predicted values After sorting, quantiles are calculated using linear interpolation to obtain confidence intervals at different confidence levels; finally, the predicted point values ​​under these confidence intervals are calculated. As the main result of energy consumption prediction.

8. The method for real-time energy consumption simulation and optimization based on AR devices in all scenarios according to claim 1, characterized in that: The collected data is standardized and preprocessed according to a hierarchical calibration mechanism. The process is as follows: First, distortion correction is performed on the visual data: Based on Zhang's calibration model, multiple images from different angles are captured, and the corner coordinates are detected using OpenCV's cornerDetect function. The least squares method is used to calculate the RGB camera intrinsic parameter matrix and distortion coefficient vector, and radial and / or tangential distortion correction is performed on the image to reduce image edges and improve the accuracy of target device shape and position measurement. Second, accurate registration of depth data and visual data is achieved: Using the focal length and image principal point coordinates in the intrinsic parameter matrix K, the pixel coordinates representing the image position and the depth values ​​representing the distance information from the pixel to the RGB camera are back-projected into a 3D point cloud, and invalid depth points are removed to generate a set of matching point pairs. Next, the initial transformation parameters are solved using OpenCV to achieve preliminary alignment between visual data and depth data, reducing the overall offset of the 3D point cloud. Finally, starting from the initial transformation parameters, the ICP algorithm of the PCL library is used to gradually reduce the distance error between the 3D point cloud and visual feature points by constructing a minimization objective function until the convergence condition is met, thus solving for the optimal transformation and achieving spatial consistency between depth data and visual data.

9. A real-time energy consumption simulation and optimization system for all scenarios based on AR devices, characterized in that: The device includes a memory and a processor. The memory includes a real-time energy consumption simulation and optimization program for the entire scene based on an AR device. When the processor executes the real-time energy consumption simulation and optimization program for the entire scene based on an AR device, it implements the real-time energy consumption simulation and optimization method for the entire scene as described in claims 1 to 8.

Images