Visual real-time dotting binding method, device and equipment based on digital twinning and medium

Abstract

This invention discloses a method, apparatus, device, and medium for real-time visualization and binding based on digital twins. The method includes: determining the coordinates of an interactive operation in response to a user's click, and generating a ray vector in the world space coordinate system based on the interactive operation coordinates; traversing the spatial index of the model object to determine at least one candidate model object that intersects with the ray vector; obtaining the spatial index of the model object through spatial analysis of geometric units; filtering each candidate model object based on the ray vector to obtain the target model object and its corresponding target virtual anchor point coordinates; generating a visualized virtual anchor point at the target virtual anchor point coordinates and determining the target binding device corresponding to the target virtual anchor point coordinates; generating and storing the binding relationship between the visualized virtual anchor point and the target binding device to achieve real-time visualization and binding between physical devices and virtual object models in a digital twin scenario.

Description

Technical Field

[0001] This invention relates to the field of digital twin and industrial internet technology, and in particular to a method, apparatus, device and medium for visual real-time point binding based on digital twin. Background Technology

[0002] In the construction of a digital twin system, point binding, which is the process of associating information such as the device identifiers of physical sensors or devices with the positions of specific virtual objects in the 3D model, is a key step in connecting the virtual and real worlds.

[0003] Traditional point binding methods typically require developers to manually look up the coordinates of the 3D model and manually input the correspondence between device identifiers and model coordinates in configuration files or databases. For large scenes containing hundreds or thousands of points, this manual method of looking up coordinates and filling in tables is prone to errors, consumes a lot of time, and is inefficient. Furthermore, the process cannot be visualized during real-time point binding, which can easily lead to misattribution (such as binding device identifier information of device X to device B), making maintenance difficult.

[0004] Therefore, how to achieve visualized real-time point binding based on digital twin scenarios, and improve the efficiency and accuracy of point binding in the process, has become an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for visual real-time point binding based on digital twins, so as to realize visual real-time point binding based on digital twin scenarios and improve the point binding efficiency and binding accuracy in the point binding process.

[0006] According to one aspect of the present invention, a method for visual real-time point binding based on digital twins is provided, the method comprising:

[0007] In response to a target user's click operation on a pre-built digital twin scene model based on a visual interactive interface, the coordinates of the target user's interaction operation are determined, and a ray vector in the world space coordinate system is generated based on the interaction operation coordinates.

[0008] The model objects are traversed using a pre-built spatial index to identify at least one candidate model object that intersects with the ray vector; the spatial index is obtained by spatial analysis of the geometric units contained in the digital twin scene model.

[0009] Based on the ray vector, the candidate model objects are filtered to obtain the target model object, and the target virtual anchor point coordinates corresponding to the target model object are obtained.

[0010] A visual virtual anchor point is generated at the target virtual anchor point coordinates, and the target binding device corresponding to the target virtual anchor point coordinates is determined;

[0011] Generate and store the binding relationship between the visualized virtual anchor points and the target bound device to realize the visualized real-time point binding between physical devices and virtual object models in a digital twin scenario.

[0012] According to another aspect of the present invention, a real-time visual dot binding device based on digital twins is provided, the device comprising:

[0013] The click operation response module is used to respond to the click operation of the target user on the pre-built digital twin scene model based on the visual interactive interface, determine the interaction operation coordinates of the target user, and generate a ray vector in the world space coordinate system based on the interaction operation coordinates.

[0014] The candidate model object determination module is used to traverse the pre-built model object spatial index and determine at least one candidate model object that has an intersection relationship with the ray vector; the model object spatial index is obtained by spatial analysis of the geometric units contained in the digital twin scene model;

[0015] The target model object determination module is used to filter each of the candidate model objects based on the ray vector to obtain the target model object, and the target virtual anchor point coordinates corresponding to the target model object;

[0016] The binding device determination module is used to generate a visual virtual anchor point at the coordinates of the target virtual anchor point and determine the target binding device corresponding to the coordinates of the target virtual anchor point;

[0017] The virtual-physical binding module is used to generate and store the binding relationship between the visualized virtual anchor points and the target binding device, so as to realize the visualized real-time point binding between physical devices and virtual object models in the digital twin scenario.

[0018] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0019] At least one processor; and

[0020] A memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the real-time visual dot binding method based on digital twins as described in any embodiment of the present invention.

[0022] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the visualization real-time dot binding method based on digital twins as described in any embodiment of the present invention.

[0023] The technical solution of this invention generates a ray vector in response to the interactive operation coordinates of the target user, and identifies at least one candidate model object that intersects with the ray vector. The candidate model objects are then filtered to obtain the target model object and its corresponding target virtual anchor point coordinates. A visual virtual anchor point is generated at the target virtual anchor point coordinates, completing the binding with the target binding device. This achieves visualized and intelligent real-time point binding between physical devices and virtual object models in a digital twin scenario, eliminating the need for manual binding and improving real-time point binding efficiency. The visualization-based intuitive selection of model objects and automatic determination of the matching target binding device further enhances point binding efficiency and accuracy.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a real-time visualization and binding method based on digital twins according to Embodiment 1 of the present invention;

[0027] Figure 2 This is a flowchart of a real-time visualization and binding method based on digital twins according to Embodiment 2 of the present invention;

[0028] Figure 3 This is a schematic diagram of a real-time visualization and binding method based on digital twins according to Embodiment 3 of the present invention;

[0029] Figure 4 This is a schematic diagram of a real-time visualization and binding device based on digital twins according to Embodiment 4 of the present invention;

[0030] Figure 5 This is a schematic diagram of the structure of an electronic device that implements the visualization real-time dot binding method based on digital twins according to embodiments of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] Example 1

[0034] Figure 1 This is a flowchart of a real-time visualization and binding method based on digital twins, provided in Embodiment 1 of the present invention. This embodiment is applicable to the real-time binding of information such as the location of virtual 3D models and the identification of physical equipment in digital twin scenarios such as smart factories, smart parks, and building automation. This method can be executed by a real-time visualization and binding device based on digital twins, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0035] S110. In response to the target user's click operation on the pre-built digital twin scene model based on the visual interactive interface, determine the target user's interaction operation coordinates, and generate a ray vector in the world space coordinate system based on the interaction operation coordinates.

[0036] S120. Traverse the pre-built model object spatial index to determine at least one candidate model object that has an intersection relationship with the ray vector; the model object spatial index is obtained by spatial analysis of the geometric units contained in the digital twin scene model.

[0037] S130. Based on the ray vector, the candidate model objects are filtered to obtain the target model object and the coordinates of the target virtual anchor point corresponding to the target model object.

[0038] S140. Generate a visual virtual anchor point at the target virtual anchor point coordinates and determine the target binding device corresponding to the target virtual anchor point coordinates.

[0039] S150. Generate and store the binding relationship between the visualized virtual anchor points and the target bound device to realize the visualized real-time point binding between physical devices and virtual object models in the digital twin scenario.

[0040] The digital twin scene model can be a 3D scene model built based on 3D (Three-Dimensional) rendering technology, such as a smart factory scene, a smart park scene, and a building automation scene. For example, the digital twin scene model can be a 3D scene model corresponding to the digital twin system of an ethylene cracking workshop.

[0041] A visual interactive interface for the digital twin scene model is pre-deployed on the front end. Using the core Web 3D library Three.js and WebGL technology, the BIM (Building Information Modeling) or 3D model of the digital twin scene model can be loaded into the visual interactive interface. Information such as the model's triangular faces, materials, and spatial coordinates can be analyzed. Based on LOD (Level of Detail) rendering technology, multiple precision versions of the model can be set, and the model can be bound to distance thresholds. This allows for dynamic switching of model precision as the camera's perspective changes, ensuring high rendering frame rates for large-scale scenes.

[0042] For example, a mapping relationship between distance and model accuracy can be established based on the Euclidean distance D between the camera and the digital twin scene model. For instance, if D < 10m, a high-precision model is loaded; if 10m ≤ D < 50m, a medium-precision model is loaded; and if d ≥ 50m, a low-precision model is loaded.

[0043] Furthermore, a full list of physical devices corresponding to the digital twin scenario model is pre-retrieved from the backend IoT platform or backend server. Metadata such as device identification information, device name, device type, region, and protocol format are extracted to build a device metadata index library. Optionally, the device metadata index library can be indexed by keywords and the first letter of the pinyin, and the index data can be cached in the frontend logic control layer to support fast local retrieval.

[0044] Furthermore, a WebWorker background thread can be used to construct a BVH index and / or an Octree spatial index, thereby avoiding blocking the front-end UI (User Interface) thread. The bounding boxes of all model objects in the digital twin scene model are calculated, generating spatial indices for each model object and caching them in the spatial index module of the front-end logic control layer for later use. For example, the spatial coordinate system of the digital twin scene model can be partitioned according to octree and / or BVH (Bounding Volume Hierarchy) rules, the bounding box coordinates of each model object can be calculated, and the bounding boxes can be associated with the object's identification information to generate a spatial index for each model object.

[0045] The target user can be any technical personnel involved in real-time tracking and binding. The target user can perform binding and clicking operations on model objects within the digital twin scene model through a visual interactive interface deployed with the digital twin scene model. Upon detecting the target user's click, the system responds by determining the screen coordinates of the user's mouse click, i.e., the interaction operation coordinates.

[0046] Based on the interaction coordinates, the inverse of the camera projection matrix is ​​used to convert them into ray vectors in the world space coordinate system. Specifically, assume the coordinates of the interaction clicked by the target user are... The screen resolution is W×H, the camera's projection matrix is ​​denoted as P, and the view matrix is ​​denoted as V. The interactive coordinates are converted to standardized device coordinates. :

[0047]

[0048]

[0049] Determine the composite inverse matrix between the projection matrix P and the view matrix V. Extend standardized equipment coordinates to homogeneous coordinates. ,and Multiplying these yields the ray origin O and ray direction D in the world space coordinate system. Here, ray origin O represents the camera position, and ray direction D is the normalized unit vector. The ray vector is expressed as follows:

[0050]

[0051] in, , where t is the step size parameter of the ray, and different values ​​of t correspond to different world space points on the ray.

[0052] The model objects are traversed using a pre-built spatial index to identify at least one candidate model object that intersects with the ray vector. Specifically, the spatial index is obtained by spatially resolving the geometric units contained in the digital twin scene model.

[0053] In one specific implementation of generating a spatial index for model objects, the geometric units of all bounding boxes to be generated in the digital twin scene model are parsed. These geometric units can be individual model objects, individual triangular faces, and model sub-parts, etc. The world coordinate system of all vertices of each geometric unit is extracted, and the AABB bounding box of each geometric unit is determined based on the world coordinate system of all vertices. Hierarchical merging and spatial partitioning are performed according to the spatial indexing rules corresponding to the BVH index and / or Octree index, respectively. For example, for the BVH index, a top-down or bottom-up approach can be used to merge the AABB bounding boxes of adjacent or spatially overlapping geometric units to generate the bounding box of the parent node, thus forming a tree hierarchy and obtaining a spatial index for model objects with hierarchical and indexed relationships.

[0054] During the traversal of model objects in the model object space index, one can start from the root node of the model object space index and determine the intersection between the bounding boxes of model objects and ray vectors at different traversed nodes. If the bounding boxes of model objects under the corresponding nodes intersect with the ray vectors, then that object model is determined as a candidate model object. Based on the intersection judgment results of the ray vectors with all model objects, at least one candidate model object can be determined.

[0055] The target model object is obtained by filtering candidate model objects based on ray vectors. For example, the coordinates of the intersection points between the ray vector and each candidate model object can be determined, and the distance between each intersection point and the starting point O of the ray vector can be determined. The candidate model object corresponding to the closest intersection point is determined as the target model object, and the coordinates of the intersection point between the ray vector and the target model object are used as the coordinates of the target virtual anchor point corresponding to the target model object.

[0056] To further improve the accuracy of determining the target model object and the coordinates of its target virtual anchor points, the target model object can be determined at a finer granular level. In one optional embodiment, the target model object and its corresponding target virtual anchor point coordinates are obtained by filtering candidate model objects based on ray vectors, including:

[0057] Step a1: For any candidate model object, traverse the triangular facets corresponding to the candidate model object based on the model object spatial index, and determine the coordinates of the reference intersection points of the ray vector and each triangular facet.

[0058] Step a2: Determine the target model object and the target intersection point coordinates corresponding to the target model object based on the coordinates of each reference intersection point corresponding to each candidate model object.

[0059] Step a3: Obtain the normal vector of the triangular facet to which the target intersection point coordinates belong, and perform offset correction on the target intersection point coordinates based on the normal vector to obtain the target virtual anchor point coordinates corresponding to the target model object.

[0060] For any candidate model object, based on the spatial index of the model object, the corresponding triangular facets are traversed. The Möller-Trumbore algorithm can be used to calculate the precise intersection points of the ray vector with each triangular facet, obtaining the reference intersection point coordinates between each triangular facet of the candidate model object and the ray vector. The reference distances between the reference intersection point coordinates of each triangular facet of the candidate model object and the starting point O of the ray vector are determined, and the shortest reference distance is selected as the target reference distance for the candidate model object. Based on the above method, the target reference distances for each candidate model object are determined, and the candidate model object with the shortest target reference distance is selected as the target model object. The coordinates of the reference intersection point of the target model object with the shortest target reference distance are then used as the target intersection point coordinates.

[0061] Determine the unit normal vector n of the triangular facet containing the target intersection point P. This can be obtained by taking the cross product of the coordinates of the three vertices of the triangular facet and normalizing it. Offset and correct the target intersection point P along the normal direction to obtain the corrected coordinates P', which are the virtual anchor point coordinates of the target. P' is determined as follows:

[0062]

[0063] in, The preset offset can be set, for example, the value range can be 0.01~0.02m, to ensure that the virtual anchor point does not overlap with the model surface.

[0064] Generate visual virtual anchor points at the target virtual anchor point coordinates. Specifically, the front-end rendering engine can use instanced rendering technology to generate visual virtual anchor points, such as colored dots or charts, at the target virtual anchor point coordinates. It should be noted that since there may be large-scale anchor point binding scenarios during real-time point binding, instanced rendering technology can draw thousands of anchor points simply by changing the transformation matrix, thereby reducing draw call overhead. Specifically, only one anchor point geometry model can be created, and a large number of anchor points can be rendered by modifying the transformation matrix. The transformation matrix can be expressed as follows:

[0065]

[0066] in, The virtual anchor point coordinates of the target. , and The scaling factor for the anchor points allows for rendering of a large number of anchor points by simply changing the transformation matrix.

[0067] Determine the target binding device corresponding to the target virtual anchor point coordinates. The target binding device can be the device to be bound to the object model at the target virtual anchor point coordinate location. In an optional embodiment, determining the target binding device corresponding to the target virtual anchor point coordinates includes:

[0068] Step b1: Determine the spatial region to which the target model object belongs, corresponding to the coordinates of the target virtual anchor point, and obtain at least one unbound device within the spatial region.

[0069] Step b2: Determine the device display priority for each unbound device based on the first spatial model corresponding to the target model object and the second spatial model corresponding to each unbound device.

[0070] Step b3: Based on the display priority of each device, display each unbound device in the visual interactive interface so that the target user can select the target bound device from the unbound devices through the visual interactive interface.

[0071] Determine the spatial region to which the target model object belongs, corresponding to the coordinates of the target virtual anchor point. It should be noted that different regions correspond to different spatial coordinate ranges, and each region includes at least one model object belonging to that region, such as walls, equipment, etc. For example, spatial coordinate ranges are divided according to building, floor, and space, such as the spatial coordinate range of the 3rd floor machine room in Building A, which is x∈[10,20], y∈[30,40], z∈[9,12].

[0072] The spatial region to which the target virtual anchor point falls is determined based on its coordinates. The target model object is the model object to be bound corresponding to the target virtual anchor point coordinates, i.e., the model object corresponding to the target bound device in the scene model. All unbound devices in this spatial region are obtained by filtering from the device metadata index. The device metadata index pre-stores all physical devices corresponding to the digital twin scene model and the device metadata corresponding to each physical device, including the spatial region to which the device belongs and the device binding status.

[0073] Understandably, once a physical device completes device binding in a digital twin scenario model, its device binding status in its device metadata will be "bound". The same spatial region may contain one or more physical devices, and each physical device has its own binding status information. After determining the spatial region to which the target model object corresponding to the target virtual anchor point coordinates belongs, at least one unbound device belonging to that spatial region with an unbound status can be obtained.

[0074] The front-end visual interaction page can display various unbound devices in a drop-down list for target users to select, and the selected unbound device will be designated as the target binding device, i.e., the device to be bound. Furthermore, the target binding device can be further narrowed down based on keyword indexing. For example, the target user can enter search keywords, and fuzzy matching will be performed based on an inverted index. For instance, if the target user enters "temperature," it can automatically match "temperature sensor," "humidity thermometer," etc., from the unbound devices, and also supports matching based on the first letter of the pinyin.

[0075] To further achieve intelligent and precise selection of target-bound devices, the display priority of each unbound device can be determined based on the first spatial model of the target object corresponding to the virtual anchor point coordinates and the second spatial models corresponding to each unbound device. For example, the first spatial model is the 3D spatial model corresponding to the target object, and the second spatial model is the 3D spatial model corresponding to each unbound device. The similarity matching degree between the first spatial model and the object spatial models corresponding to each second spatial model is determined. For example, similarity matching can be performed based on multi-dimensional features such as volume, surface area, centroid coordinates, geometric shape, and size to obtain the model similarity between the first spatial model and each second spatial model. Based on the model similarity, the models are prioritized from high to low to obtain the display priority of each unbound device.

[0076] Based on the display priority of each device, they are presented in a list from highest to lowest in the visual interactive interface for target users to refer to. Target users can intuitively see the unbound devices that best match the spatial model according to the display order, thus more quickly and accurately locating the target device to be bound.

[0077] Furthermore, to improve the accuracy of spatial feature matching, during the spatial model similarity matching process, a pre-trained spatial similarity matching model can be used to perform similarity matching between the first spatial model and the second spatial models. For example, the first spatial model and each of the second spatial models can be input into the pre-trained spatial similarity matching model to obtain a similarity score between the first spatial model and each of the second spatial models. A higher score indicates a higher similarity between the two spatial models; a lower score indicates a lower similarity.

[0078] The spatial similarity matching model can be trained using a large number of object models from digital twin scene models as sample data, with the similarity between object models serving as sample labels. The spatial similarity matching model can be obtained by training a pre-selected network model based on the sample data. The network model can be, for example, a convolutional neural network, but this embodiment does not impose any limitations on it.

[0079] Generate and store the binding relationship between visual virtual anchor points and target bound devices to achieve real-time visual point binding between physical devices and virtual object models in digital twin scenarios. Specifically, a unique UUID (Universal Unique Identifier) ​​can be generated for the visual virtual anchor points. This unique UUID identifier, i.e., the device ID, can be generated based on a UUID algorithm using timestamps and random numbers.

[0080] The device ID is mapped to a binary bit in the bitmap. For example, device ID=101 corresponds to the 101st bit in the bitmap. After successful binding, this bit is set to 1, and 0 is set to 0 if not bound. O(1) time complexity duplicate binding detection is achieved through bit-by-bit lookup in the bitmap. The binding relationship can include the visual virtual anchor UUID, the target virtual anchor coordinates P'(x,y,z), the normal vector n, and the device name, etc. The binding relationship record can be synchronized to the backend database and cached on the frontend for subsequent querying, selection, and retrieval.

[0081] The technical solution of this invention generates a ray vector in response to the interactive operation coordinates of the target user, and identifies at least one candidate model object that intersects with the ray vector. The candidate model objects are then filtered to obtain the target model object and its corresponding target virtual anchor point coordinates. A visual virtual anchor point is generated at the target virtual anchor point coordinates, completing the binding with the target binding device. This achieves visualized and intelligent real-time point binding between physical devices and virtual object models in a digital twin scenario, eliminating the need for manual binding and improving real-time point binding efficiency. The visualization-based intuitive selection of model objects and automatic determination of the matching target binding device further enhances point binding efficiency and accuracy.

[0082] Example 2

[0083] Figure 2 This is a flowchart of a real-time visualization and binding method based on digital twins provided in Embodiment 2 of the present invention. This embodiment has been optimized and improved based on the above technical solutions.

[0084] Furthermore, after the step "Generate and store the binding relationship between the visualized virtual anchor point and the target bound device", the following step is added: "When it is detected that the device status of the target bound device needs to be updated, obtain the current timestamp of the current time period and the historical state rendering update timestamp of the target bound device in the historical time period; determine the throttling time threshold according to the rendering refresh rate corresponding to the target bound device; based on the current timestamp and the historical state rendering update timestamp, determine whether to update the device status of the target bound device based on the throttling time threshold; if so, update the device status of the target bound device according to the obtained real-time device status of the target bound device." This improves the method for updating the status of the target bound device.

[0085] It should be noted that for parts not described in detail in the embodiments of the present invention, please refer to the descriptions in other embodiments. For example... Figure 2 As shown, the method includes the following specific steps:

[0086] S210. In response to the target user's click operation on the pre-built digital twin scene model based on the visual interactive interface, determine the target user's interaction operation coordinates, and generate a ray vector in the world space coordinate system based on the interaction operation coordinates.

[0087] S220. Traverse the pre-built model object spatial index to determine at least one candidate model object that has an intersection relationship with the ray vector; the model object spatial index is obtained by spatial analysis of the geometric units contained in the digital twin scene model.

[0088] S230. Based on the ray vector, the candidate model objects are filtered to obtain the target model object and the coordinates of the target virtual anchor point corresponding to the target model object.

[0089] S240. Generate a visual virtual anchor point at the target virtual anchor point coordinates and determine the target binding device corresponding to the target virtual anchor point coordinates.

[0090] S250. Generate and store the binding relationship between the visualized virtual anchor points and the target bound device to realize the visualized real-time point binding between physical devices and virtual object models in the digital twin scenario.

[0091] S260. When it is detected that the device status of the target bound device needs to be updated, obtain the current timestamp of the current time period and the historical state rendering update timestamp of the target bound device in the historical time period.

[0092] S270. Determine the throttling time threshold based on the rendering refresh rate corresponding to the target bound device.

[0093] S280. Render update timestamp based on current timestamp and historical status, and determine whether to update device status for target bound device based on throttling time threshold.

[0094] S290. If so, then update the target bound device status based on the real-time device status of the obtained target bound device.

[0095] Specifically, a persistent long-lived connection can be established between the front-end WebSocket client and the back-end server for data transmission, with a pre-agreed data transmission format, such as Protbuf (Protocol Buffers) or JSON Patch (JSOM) format. Preferably, when transmitting device data, such as device status data, binary serialization can be performed according to the preset Protobuf protocol format, which can compress the data volume by about 60% compared to JSON.

[0096] The device status can include current device data and alarm status. For example, current device data can be temperature data and liquid level data at the current time. Different physical devices have different device data; for example, the device data for a temperature sensor is the temperature value, and the device data for a liquid level sensor is the liquid level value. Alarm status can be one of four states: normal, offline, warning, and alarm.

[0097] Specifically, the backend server can monitor the device status of physical devices and feed the monitoring results back to the frontend. When it detects that the device status of the target bound device needs to be updated, it can obtain the current device data for the current time period. It should be noted that because the data update frequency is high, such as the vibration sensor sampling frequency of 100Hz, the frontend rendering refresh burden is large. Therefore, in order to limit the frontend rendering refresh rate and avoid UI thread blocking, a throttling strategy can be used for frontend rendering.

[0098] Furthermore, during the data transmission process between the front-end and back-end, only the changed part of the data, i.e. the incremental part, can be transmitted. For example, if the liquid level changes from 50% to 80%, only the changed value of 80% can be transmitted, thus avoiding data transmission congestion and low transmission efficiency caused by transmitting the full amount of data.

[0099] For example, the real-time data of the target bound device can be pre-written into a thread pool. Device status update requirements can be based on updated device data; that is, when real-time updated data is obtained, the relevant data can be displayed and written into the created thread pool. Simultaneously, the current timestamp of the current time period and the historical state rendering update timestamps of the target bound device in the historical time period are obtained. The historical time period is the rendering period of the last data update, and the time corresponding to the previous data update rendering period is the historical state rendering update timestamp.

[0100] The throttling time threshold is determined based on the rendering refresh rate of the target bound device. For example, if the rendering refresh rate F is set to 30fps, then the throttling time threshold T = 1 / F = 33.3ms. Assume the current timestamp is... The historical state rendering update timestamp is ,like Then, the device state of the target bound device is updated to ensure that the UI thread is not blocked by high-frequency data.

[0101] The target bound device status is updated based on the real-time device status obtained. For example, if the real-time device status is 80% and the historical device status is 50%, the current device status of the target bound device is updated to 80% and displayed in the visualization interface for the target user to view.

[0102] Furthermore, when the backend server's point state machine detects an alarm status update for the target bound device, it updates the alarm status and renders it on the front-end interface. Four alarm statuses are set for each virtual anchor point: normal, offline, warning, and alarm. If the device's real-time data value V exceeds the corresponding warning threshold for that device... If the real-time data value V of the device exceeds the corresponding alarm threshold, then the alarm status of the device is determined to be a warning; If the device reports no data, its alarm status is determined to be "alarm"; if the device reports no data, its alarm status is "offline"; if the device's real-time data V does not exceed the corresponding warning threshold... If so, the device alarm status is determined to be normal.

[0103] Optionally, the front-end rendering engine can dynamically switch the 3D materials and animation effects of the corresponding virtual anchor points based on the current device status of the target bound device. For example, a solid green light in normal status, a semi-transparent gray light in offline status, a flashing yellow light in warning status, and a red breathing light effect in alarm status. The device status data of the target bound device is synchronized with the binding record to the back-end database, and status backtracking is supported.

[0104] The above technical solution significantly reduces network bandwidth consumption and data parsing latency in high-concurrency scenarios by using WebSocket long connections combined with incremental data transmission via Protobuf and / or JSON Patch. It achieves precise traffic control through throttling strategies and frame rate limiting, avoiding UI thread blocking and ensuring smooth rendering in large-scale anchor point scenarios. Furthermore, it enables precise state linkage between physical devices and virtual anchor points through state machine and device state mapping, ensuring determinism and traceability of state transitions. Finally, it provides intuitive and efficient visual feedback to target users through dynamic switching of 3D materials and animation effects, significantly improving the real-time performance, stability, and operational efficiency of point binding in digital twin scenarios, making it more suitable for industrial-grade high-concurrency scenarios.

[0105] Furthermore, this embodiment also provides a data consistency synchronization method under scenarios of abnormal front-end and back-end connections. In an optional embodiment, after generating and storing the binding relationship between the visualized virtual anchor point and the target bound device, the method further includes:

[0106] Step c1: When a communication connection failure is detected between the backend server used to obtain the real-time device status of the target bound device, obtain the number of reconnection attempts with the backend server in the current time period.

[0107] Step c2: Determine the reconnection interval period based on the number of reconnections and the preset basic reconnection time interval, and perform reconnection operations with the backend server based on the reconnection interval period.

[0108] Step c3: After the reconnection operation is successfully executed, obtain the full device status of the target bound device and construct the first Merkle tree hash value, and obtain the second Merkle tree hash value from the backend server.

[0109] Step c4: If the hash value of the first Merkle tree is inconsistent with the hash value of the second Merkle tree, then the task of fully synchronizing the device status data of the target bound device is triggered.

[0110] Specifically, the front-end WebSocket client can periodically send heartbeat packets to the back-end server, such as every 5 seconds, to check the connection status. If the back-end server does not respond with a heartbeat packet, it can be determined that the communication connection with the back-end server has been broken, triggering the calculation of the number of reconnection attempts (n) with the back-end server in the current time period. The initial value of n is 0. Based on the number of reconnection attempts (n) and the preset basic reconnection time interval... Determine the reconnection interval period :

[0111]

[0112] Among them, the basic reconnection time interval The value can be 1 second. This represents the retreat coefficient, which can take the value 2.

[0113] Set maximum interval ,like You can set it to 30 seconds to avoid infinite waiting. For example, when n=0, T=1s; when n=1, T=2s; and when n=5, T=32s, then we take 30s.

[0114] When the reconnection interval is met, a reconnection operation is performed with the backend server. After the reconnection operation is successful, the full device state of the target bound device is obtained, and a first Merkle tree hash value is constructed. A second Merkle tree hash value is then obtained from the backend server. The backend server constructs a Merkle tree for device state, which divides the full device state data into blocks by device ID, calculates the hash value of each block (e.g., using the SHA-256 algorithm), and uses these blocks as leaf nodes of the Merkle tree. The hash values ​​of parent nodes are calculated layer by layer upwards to obtain the root hash value, which is the second Merkle tree hash value. The frontend calculates the root hash of the locally cached device state Merkle tree, which is the first Merkle tree hash value.

[0115] If the first Merkle tree hash value is inconsistent with the second Merkle tree hash value, a full synchronization task of device status data for the target bound device is triggered; if the first Merkle tree hash value is consistent with the second Merkle tree hash value, no full data synchronization is required. The full synchronization task of device status data immediately sends a full status query request to the backend server, retrieves all device status data, updates the frontend device status mapping table, and corrects the display status of the virtual anchor point corresponding to the target bound device.

[0116] The above technical solution employs an exponential backoff reconnection time interval calculation method, which makes the reconnection interval grow exponentially with the number of failures and limits the maximum value. This avoids network storms and server overload caused by a large number of reconnection requests in a short period of time, while reserving sufficient time for network link recovery. This improves the reconnection success rate and effectively reduces the load on terminals and servers, ensuring the robustness and stability of the system in complex industrial network environments and achieving efficient and imperceptible self-healing in network anomaly scenarios.

[0117] Furthermore, by combining Merkle tree hash comparison with full data synchronization, the data verification method leverages the hierarchical hash aggregation characteristics of the Merkle tree to achieve consistency determination of large-scale device data simply by comparing the root hash value, reducing the verification time complexity from O(n) to O(1), significantly improving verification efficiency in high-concurrency scenarios. Moreover, the hierarchical structure of the Merkle tree allows for precise location of data inconsistencies, enabling differentiated data synchronization and effectively reducing network transmission overhead in industrial weak network environments.

[0118] It should be noted that for dynamic devices in the digital twin scenario model, such as AGVs (Automated Guided Vehicles) with positioning functions, their positions may shift. Therefore, this embodiment also provides a method for correcting the position offset of devices that may shift. In an optional embodiment, after generating and storing the binding relationship between the visualized virtual anchor point and the target bound device, the method further includes:

[0119] Step d1: Determine the device type of the target bound device. If the device type of the target bound device meets the preset dynamic movement type judgment conditions, then obtain the real-time positioning coordinates of the target bound device when the dynamic monitoring time period is met.

[0120] Step d2: Determine the spatial deviation distance of the device based on the real-time positioning coordinates of the target bound device and the coordinates of the target virtual anchor point.

[0121] Step d3: If the spatial deviation distance of the device is greater than the preset spatial deviation distance threshold, then the position drift warning task of the target bound device is triggered and executed.

[0122] Among them, the dynamic movement type judgment condition can be that the device type meets the preset dynamic device type, and the dynamic device type is a high-frequency movement device, such as AGV.

[0123] If the target bound device is a high-frequency mobile device, then the real-time positioning coordinates of the target bound device are obtained when the dynamic monitoring time period is met. The dynamic monitoring time period can be the monitoring time period for the device type to which the target device belongs. Different device types have different dynamic monitoring time periods; for example, the monitoring time period for AGV devices can be 2 hours. This embodiment does not impose any restrictions on this.

[0124] Based on the real-time location coordinates of the target bound device and the target virtual anchor point coordinates Determine the equipment spatial deviation distance D:

[0125]

[0126] If the spatial deviation distance D of the equipment is greater than the preset spatial deviation distance threshold This triggers and executes a position drift warning task for the target bound device. For example, in the 3D interface of the front-end visualization layer, the virtual anchor point is marked with an abnormal position, such as a flashing red border, and a calibration prompt pops up in the visual interactive interface to guide the target user to manually correct the anchor point coordinates. Alternatively, it can be based on real-time positioning coordinates. Automatic correction.

[0127] The above technical solution performs precise quantitative calculation of spatial deviation by determining the distance between the coordinates of the virtual anchor point and the real-time positioning coordinates of the physical device. Combined with a configurable position deviation threshold, it realizes automated and real-time early warning of dynamic device position drift, eliminates subjective judgment errors, and accurately ensures the positional consistency between the virtual anchor point and the physical device.

[0128] Understandably, in digital twin scenarios, there may be high-frequency mobile devices such as AGVs and mobile robots. When the device information of high-frequency mobile devices changes, the spatial index of the model object needs to be reconstructed in its entirety, which is time-consuming and inefficient, and cannot support the real-time dynamic point marking requirements.

[0129] Therefore, to avoid a full update of the model object spatial index, in one optional embodiment, the digital twin scene model includes a static scene region and a dynamic scene region; the static scene region contains the geometric unit of at least one static object; the dynamic scene region contains the geometric unit of at least one dynamic object; the model object spatial index includes the octree index corresponding to the static scene region and the boundary volume hierarchy (BVH) index corresponding to the dynamic scene region.

[0130] The digital twin scene model is divided into static areas, such as fixed equipment and building structures, and dynamic areas, such as AGVs and mobile robots. The static area uses an Octree index, which is structured and offers fast querying. The dynamic area uses a BVH index, which is flexible and updates quickly. These two index structures can be linked through a global spatial hash table, providing a unified query interface.

[0131] A movement frequency threshold can be set for each dynamic device. When the movement distance of a device exceeds the threshold, only the bounding box of its BVH node is updated, rather than the entire index is reconstructed. For low-frequency mobile devices, the index is automatically downgraded to a static region index, thereby reducing update overhead. In dynamic scenarios, the index update time can be reduced from seconds to milliseconds, and the ray vector detection response time of high-frequency mobile devices can be stabilized within 5ms, making it more suitable for dynamic scenarios such as AGV scheduling and robot inspection in industrial sites.

[0132] Example 3

[0133] Figure 3 This is a schematic diagram of the process structure of a real-time visualization and binding method based on digital twins, provided in Embodiment 3 of the present invention. This embodiment provides a preferred example based on the above embodiments.

[0134] The specific implementation steps for the real-time visualization and binding process based on digital twins can be as follows:

[0135] S31. In response to the target user's click operation on the pre-built digital twin scene model based on the visual interactive interface, determine the target user's interaction operation coordinates, and generate a ray vector in the world space coordinate system based on the interaction operation coordinates.

[0136] S32. Traverse the pre-built model object spatial index to determine at least one candidate model object that intersects with the ray vector; the model object spatial index is obtained by spatial analysis of the geometric units contained in the digital twin scene model.

[0137] S33. For any candidate model object, traverse the triangular facets corresponding to the candidate model object based on the spatial index of the model object, and determine the coordinates of the reference intersection points of the ray vector and each triangular facet.

[0138] S34. Determine the target model object and the target intersection point coordinates corresponding to the target model object based on the coordinates of each reference intersection point corresponding to each candidate model object.

[0139] S35. Obtain the normal vector of the triangular facet to which the target intersection point coordinates belong, and perform offset correction on the target intersection point coordinates based on the normal vector to obtain the target virtual anchor point coordinates corresponding to the target model object.

[0140] S36. Generate a visual virtual anchor point at the target virtual anchor point coordinates, determine the spatial region to which the target model object corresponding to the target virtual anchor point coordinates belongs, and obtain at least one unbound device within the spatial region.

[0141] S37. Based on the first spatial model corresponding to the target model object and the second spatial model corresponding to each unbound device, determine the device display priority corresponding to each unbound device.

[0142] S38. Based on the display priority of each device, display each unbound device in the visual interactive interface so that the target user can select the target bound device from the unbound devices based on the visual interactive interface.

[0143] S39. Generate and store the binding relationship between the visualized virtual anchor points and the target bound device to realize the visualized real-time point binding between physical devices and virtual object models in the digital twin scenario.

[0144] The implementation method for updating the device status of the target bound device after point binding can be as follows: When it is detected that the device status of the target bound device needs to be updated, obtain the current timestamp of the current time period and the historical state rendering update timestamp of the target bound device in the historical time period; determine the throttling time threshold according to the rendering refresh rate corresponding to the target bound device; based on the current timestamp and the historical state rendering update timestamp, and the throttling time threshold, determine whether to update the device status of the target bound device; if so, update the device status of the target bound device according to the obtained real-time device status of the target bound device.

[0145] The implementation of the anomaly detection and self-healing mechanism for the target bound device after point binding can be as follows: When a communication connection failure is detected between the target bound device and the backend server used to obtain the real-time device status of the target bound device, the number of reconnections with the backend server in the current time period is obtained; based on the number of reconnections and the preset basic reconnection time interval, the reconnection interval time period is determined, and a reconnection operation is performed with the backend server based on the reconnection interval time period; after the reconnection operation is successfully executed, the full device status of the target bound device is obtained and a first Merkle tree hash value is constructed, and a second Merkle tree hash value is obtained from the backend server; if the first Merkle tree hash value and the second Merkle tree hash value are inconsistent, the full synchronization task of the device status data of the target bound device is triggered.

[0146] The method for updating the device position offset of the target bound device after point binding can be as follows: determine the device type of the target bound device; if the device type of the target bound device meets the preset dynamic movement type judgment condition, then obtain the real-time positioning coordinates of the target bound device when the dynamic monitoring time period is met; determine the device spatial deviation distance based on the real-time positioning coordinates of the target bound device and the target virtual anchor point coordinates; if the device spatial deviation distance is greater than the preset spatial deviation distance threshold, then trigger and execute the target bound device position drift warning task.

[0147] The advantages of the above technical solution are as follows:

[0148] Through BVH spatial indexing and instantiated rendering technology, the system can smoothly support the calibration and rendering of tens of thousands of points in a single scene, solving the stuttering problem of traditional methods in large scenes. The introduction of triangle-level collision detection and normal offset correction ensures that virtual points accurately fit the model surface, avoiding visual errors such as "hanging" or "trapped" in the model. An intelligent recommendation algorithm based on spatial neighborhood significantly reduces the time users spend searching through massive device lists, improving binding efficiency by more than 3 times compared to traditional search methods. Front-end message throttling and incremental update mechanisms enable the system to stably handle concurrent data reporting from thousands of devices without frame drops. Built-in heartbeat reconnection and data consistency verification mechanisms ensure the accuracy and stability of data under network fluctuations or long-term operation.

[0149] In a specific application scenario, the digital twin scenario is a digital twin system for an ethylene cracking plant, which includes a lightweight BIM model with 5 million triangular facets and 12,000 physical devices, including mobile devices such as AGVs and static devices such as pressure gauges and sensors. The specific implementation steps are as follows:

[0150] Step 1: The system loads the park's lightweight BIM model, which contains 5 million triangles. An octree index for the scene is built using a WebWorker background thread, taking approximately 1.5 seconds.

[0151] Step 2: Preload 12,000 device metadata records and build a prefix search index based on a Trie tree.

[0152] Step 3: The implementer performs continuous click operations on the "ethylene cracking workshop" model. The system uses octree accelerated ray detection, and the response time of each click is controlled within 5ms.

[0153] Step 4: After clicking on the device surface, the system automatically offsets outward by 2cm along the normal to generate anchor points, avoiding intersecting with the device model.

[0154] Step 5: When a pressure gauge anchor point is selected, the system identifies that the coordinates are located in the "Cracking Furnace F-101" area. The recommended list automatically prioritizes unbound pressure sensors related to "F-101" (such as PI-10101, PI-10102).

[0155] Step 6: The user directly clicks PI-10101 to complete the binding process, requiring only two clicks and no keyboard input. After binding, real-time data streaming is enabled. The backend pushes approximately 500 changing data entries per second via WebSocket. The frontend message throttling limit restricts the rendering update frequency to 30Hz.

[0156] Step 7: When the liquid level in the "T-200" storage tank exceeds the limit, the corresponding virtual point state machine immediately switches to the "Alarm" state, triggering a red highlight flashing animation. Testing showed that in the Chrome browser, with 12,000 points rendered simultaneously, the FPS remained stable above 55 frames per second, and memory usage was kept below 800MB.

[0157] Step 8: Simulate a 30-second network interruption followed by recovery; the WebSocket client automatically reconnects. If 5 points are found to be out of sync during this period, a full status query request is immediately initiated, and the displayed status is automatically corrected.

[0158] Example 4

[0159] Figure 4 This is a schematic diagram of a real-time visualization and binding device based on digital twins, provided in Embodiment 4 of the present invention. The real-time visualization and binding device based on digital twins provided in this embodiment of the present invention is applicable to real-time binding of information such as the location of virtual 3D models and the identification of physical equipment in digital twin scenarios such as smart factories, smart parks, and building automation. This real-time visualization and binding device based on digital twins can be implemented in hardware and / or software, such as... Figure 4 As shown, the device includes: a click operation response module 401, a candidate model object determination module 402, a target model object determination module 403, a binding device determination module 404, and a virtual-real binding module 405. Among them,

[0160] Click operation response module 401 is used to respond to the click operation of the target user on the pre-built digital twin scene model based on the visual interactive interface, determine the interaction operation coordinates of the target user, and generate a ray vector in the world space coordinate system based on the interaction operation coordinates.

[0161] The candidate model object determination module 402 is used to traverse the pre-built model object spatial index and determine at least one candidate model object that has an intersection relationship with the ray vector; the model object spatial index is obtained by spatial analysis of the geometric units contained in the digital twin scene model;

[0162] The target model object determination module 403 is used to filter each of the candidate model objects based on the ray vector to obtain the target model object and the target virtual anchor point coordinates corresponding to the target model object;

[0163] The binding device determination module 404 is used to generate a visual virtual anchor point at the target virtual anchor point coordinates and determine the target binding device corresponding to the target virtual anchor point coordinates;

[0164] The virtual-physical binding module 405 is used to generate and store the binding relationship between the visualized virtual anchor point and the target binding device, so as to realize the visualized real-time point binding between physical devices and virtual object models in the digital twin scenario.

[0165] The technical solution of this invention generates a ray vector in response to the interactive operation coordinates of the target user, and identifies at least one candidate model object that intersects with the ray vector. The candidate model objects are then filtered to obtain the target model object and its corresponding target virtual anchor point coordinates. A visual virtual anchor point is generated at the target virtual anchor point coordinates, completing the binding with the target binding device. This achieves visualized and intelligent real-time point binding between physical devices and virtual object models in a digital twin scenario, eliminating the need for manual binding and improving real-time point binding efficiency. The visualization-based intuitive selection of model objects and automatic determination of the matching target binding device further enhances point binding efficiency and accuracy.

[0166] Optionally, the target model object determination module 403 is specifically used for:

[0167] For any candidate model object, the triangular facets corresponding to the candidate model object are traversed based on the spatial index of the model object to determine the coordinates of the reference intersection point between the ray vector and each of the triangular facets;

[0168] Based on the coordinates of the reference intersection points corresponding to each of the candidate model objects, the target model object and the coordinates of the target intersection point corresponding to the target model object are determined.

[0169] Obtain the normal vector of the triangular facet to which the target intersection point coordinates belong, and perform offset correction on the target intersection point coordinates based on the normal vector to obtain the target virtual anchor point coordinates corresponding to the target model object.

[0170] Optionally, the binding device determination module 404 is specifically used for:

[0171] Determine the spatial region to which the target model object belongs, corresponding to the coordinates of the target virtual anchor point, and obtain at least one unbound device within the spatial region;

[0172] Based on the first spatial model corresponding to the target model object and the second spatial model corresponding to each of the unbound devices, the device display priority corresponding to each of the unbound devices is determined.

[0173] Based on the display priority of each device, each unbound device is displayed on the visual interactive interface, so that the target user can select a target bound device from the unbound devices based on the visual interactive interface.

[0174] Optionally, the device further includes:

[0175] The timestamp acquisition module is used to acquire the current timestamp of the current time period and the historical state rendering update timestamp of the target bound device in the historical time period when it is detected that the device status of the target bound device needs to be updated after the binding relationship between the visual virtual anchor point and the target bound device is generated and stored.

[0176] The throttling time threshold determination module is used to determine the throttling time threshold based on the rendering refresh rate corresponding to the target bound device;

[0177] The status update judgment module is used to determine whether to update the device status of the target bound device based on the current timestamp and the historical status rendering update timestamp, and the throttling time threshold.

[0178] The status update module is used to update the status of the target bound device based on the real-time device status of the target bound device if a device status update is required for the target bound device.

[0179] Optionally, the device further includes:

[0180] The reconnection count acquisition module is used to acquire the number of reconnections with the backend server in the current time period after the binding relationship between the visual virtual anchor point and the target bound device is generated and stored, when the communication connection between the backend server used to acquire the real-time device status of the target bound device is detected to be failed.

[0181] The reconnection interval period determination module is used to determine the reconnection interval period based on the number of reconnections and the preset basic reconnection time interval, and to perform a reconnection operation with the backend server based on the reconnection interval period.

[0182] The hash value determination module is used to obtain the full device status of the target bound device and construct the first Merkle tree hash value after the reconnection operation is successfully executed, and obtain the second Merkle tree hash value from the backend server.

[0183] The synchronization task execution module is used to trigger a full synchronization task of device status data of the target bound device if the first Merkle tree hash value is inconsistent with the second Merkle tree hash value.

[0184] Optionally, the device further includes:

[0185] The device type determination module is used to determine the device type of the target bound device after generating and storing the binding relationship between the visual virtual anchor point and the target bound device. If the device type of the target bound device meets the preset dynamic movement type judgment condition, the real-time positioning coordinates of the target bound device are obtained when the dynamic monitoring time period is met.

[0186] The spatial deviation distance determination module is used to determine the spatial deviation distance of the device based on the real-time positioning coordinates of the target bound device and the coordinates of the target virtual anchor point;

[0187] The drift warning task triggering module is used to trigger and execute the position drift warning task of the target bound device if the spatial deviation distance of the device is greater than a preset spatial deviation distance threshold.

[0188] Optionally, the digital twin scene model includes a static scene region and a dynamic scene region; the static scene region contains geometric units of at least one static object; the dynamic scene region contains geometric units of at least one dynamic object; the model object space index includes an octree index corresponding to the static scene region and a boundary volume hierarchy (BVH) index corresponding to the dynamic scene region.

[0189] The visualization real-time dot binding device based on digital twins provided in this embodiment of the invention can execute the visualization real-time dot binding method based on digital twins provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0190] Example 5

[0191] Figure 5A schematic diagram of an electronic device 50 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0192] like Figure 5 As shown, the electronic device 50 includes at least one processor 51 and a memory, such as a read-only memory (ROM) 52 and a random access memory (RAM) 53, communicatively connected to the at least one processor 51. The memory stores computer programs executable by the at least one processor. The processor 51 can perform various appropriate actions and processes based on the computer program stored in the ROM 52 or loaded from storage unit 58 into the RAM 53. The RAM 53 can also store various programs and data required for the operation of the electronic device 50. The processor 51, ROM 52, and RAM 53 are interconnected via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0193] Multiple components in electronic device 50 are connected to I / O interface 55, including: input unit 56, such as keyboard, mouse, etc.; output unit 57, such as various types of monitors, speakers, etc.; storage unit 58, such as disk, optical disk, etc.; and communication unit 59, such as network card, modem, wireless transceiver, etc. Communication unit 59 allows electronic device 50 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0194] Processor 51 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 51 performs the various methods and processes described above, such as a real-time dot binding method for visualization based on digital twins.

[0195] In some embodiments, the digital twin-based visual real-time dot binding method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 58. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 50 via ROM 52 and / or communication unit 59. When the computer program is loaded into RAM 53 and executed by processor 51, one or more steps of the digital twin-based visual real-time dot binding method described above can be performed. Alternatively, in other embodiments, processor 51 can be configured to perform the digital twin-based visual real-time dot binding method by any other suitable means (e.g., by means of firmware).

[0196] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0197] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0198] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0199] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0200] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0201] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0202] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0203] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for visual real-time point binding based on digital twins, characterized in that, include: In response to a target user's click operation on a pre-built digital twin scene model based on a visual interactive interface, the coordinates of the target user's interaction operation are determined, and a ray vector in the world space coordinate system is generated based on the interaction operation coordinates. The model objects are traversed using a pre-built spatial index to identify at least one candidate model object that intersects with the ray vector; the spatial index is obtained by spatial analysis of the geometric units contained in the digital twin scene model. Based on the ray vector, the candidate model objects are filtered to obtain the target model object, and the target virtual anchor point coordinates corresponding to the target model object are obtained. A visual virtual anchor point is generated at the target virtual anchor point coordinates, and the target binding device corresponding to the target virtual anchor point coordinates is determined; Generate and store the binding relationship between the visualized virtual anchor points and the target bound device to realize the visualized real-time point binding between physical devices and virtual object models in a digital twin scenario.

2. The method according to claim 1, characterized in that, The step of filtering each candidate model object based on the ray vector to obtain the target model object, and the target virtual anchor point coordinates corresponding to the target model object, includes: For any candidate model object, the triangular facets corresponding to the candidate model object are traversed based on the spatial index of the model object to determine the coordinates of the reference intersection point between the ray vector and each of the triangular facets; Based on the coordinates of the reference intersection points corresponding to each of the candidate model objects, the target model object and the coordinates of the target intersection point corresponding to the target model object are determined. Obtain the normal vector of the triangular facet to which the target intersection point coordinates belong, and perform offset correction on the target intersection point coordinates based on the normal vector to obtain the target virtual anchor point coordinates corresponding to the target model object.

3. The method according to claim 1, characterized in that, The step of determining the target binding device corresponding to the target virtual anchor point coordinates includes: Determine the spatial region to which the target model object belongs, corresponding to the coordinates of the target virtual anchor point, and obtain at least one unbound device within the spatial region; Based on the first spatial model corresponding to the target model object and the second spatial model corresponding to each of the unbound devices, the device display priority corresponding to each of the unbound devices is determined. Based on the display priority of each device, each unbound device is displayed on the visual interactive interface, so that the target user can select a target bound device from the unbound devices based on the visual interactive interface.

4. The method according to claim 1, characterized in that, After generating and storing the binding relationship between the visualized virtual anchor point and the target binding device, the method further includes: When it is detected that the device status of the target bound device needs to be updated, the current timestamp of the current time period and the historical state rendering update timestamp of the target bound device in the historical time period are obtained. The throttling time threshold is determined based on the rendering refresh rate corresponding to the target bound device; Based on the current timestamp and the historical state rendering update timestamp, and based on the throttling time threshold, determine whether to update the device status of the target bound device; If so, the target bound device status is updated based on the real-time device status of the obtained target bound device.

5. The method according to claim 4, characterized in that, After generating and storing the binding relationship between the visualized virtual anchor point and the target binding device, the method further includes: When a communication connection failure is detected between the backend server used to obtain the real-time device status of the target bound device, the number of reconnection attempts with the backend server in the current time period is obtained. Based on the number of reconnections and the preset basic reconnection time interval, a reconnection interval period is determined, and a reconnection operation is performed with the backend server based on the reconnection interval period. After the reconnection operation is successfully executed, the full device status of the target bound device is obtained and a first Merkle tree hash value is constructed, and a second Merkle tree hash value is obtained from the backend server; If the first Merkle tree hash value is inconsistent with the second Merkle tree hash value, a full synchronization task of device status data of the target bound device will be triggered.

6. The method according to claim 1, characterized in that, After generating and storing the binding relationship between the visualized virtual anchor point and the target binding device, the method further includes: The device type of the target bound device is determined. If the device type of the target bound device meets the preset dynamic movement type judgment condition, the real-time positioning coordinates of the target bound device are obtained when the dynamic monitoring time period is met. The spatial deviation distance of the device is determined based on the real-time positioning coordinates of the target bound device and the coordinates of the target virtual anchor point; If the spatial deviation distance of the device is greater than the preset spatial deviation distance threshold, the position drift warning task of the target bound device is triggered and executed.

7. The method according to claim 1, characterized in that, The digital twin scene model includes a static scene region and a dynamic scene region; the static scene region contains the geometric unit of at least one static object; the dynamic scene region contains the geometric unit of at least one dynamic object; the model object space index includes the octree index corresponding to the static scene region and the boundary volume hierarchy (BVH) index corresponding to the dynamic scene region.

8. A real-time visual tracking and binding device based on digital twins, characterized in that, include: The click operation response module is used to respond to the click operation of the target user on the pre-built digital twin scene model based on the visual interactive interface, determine the interaction operation coordinates of the target user, and generate a ray vector in the world space coordinate system based on the interaction operation coordinates. The candidate model object determination module is used to traverse the pre-built model object spatial index and determine at least one candidate model object that has an intersection relationship with the ray vector; the model object spatial index is obtained by spatial analysis of the geometric units contained in the digital twin scene model; The target model object determination module is used to filter each of the candidate model objects based on the ray vector to obtain the target model object, and the target virtual anchor point coordinates corresponding to the target model object; The binding device determination module is used to generate a visual virtual anchor point at the coordinates of the target virtual anchor point and determine the target binding device corresponding to the coordinates of the target virtual anchor point; The virtual-physical binding module is used to generate and store the binding relationship between the visualized virtual anchor points and the target binding device, so as to realize the visualized real-time point binding between physical devices and virtual object models in the digital twin scenario.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the visual real-time dot binding method based on digital twins as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the real-time visualization and dot binding method based on digital twins as described in any one of claims 1-7.

Images