Method for Using BIM Data on Mobile Devices through a Graphic Interaction Engine

Abstract

The present invention relates to the field of computers. A method for using BIM data on a mobile device through a graphics interaction engine, based on the rules set by drawing tags and BIM, establishes a drawing jump link with a recommended drawing list A and a corresponding recommended weight vector Va; based on the deep learning of user usage data, generates a recommended drawing list B for the drawing jump link and a corresponding recommended weight vector Vb; the client performs clipping and normalization processing on the recommended weight vector Va and the recommended weight vector Vb, and sorts the recommended weight vectors in descending order of weight to obtain the final recommended drawing list; when the user opens a drawing, the client provides the corresponding drawing link to the user according to the final recommended drawing list. Through the optimization of the jump link, the present invention establishes a link relationship between the BIM three-dimensional model, the BIM two-dimensional drawing and the non-BIM three-dimensional model, the non-BIM two-dimensional drawing.

Description

Technical Field

[0001] The present invention relates to the field of computers, and particularly to a method for computer data processing. Background Art

[0002] When using a computer to read or process drawings, the following problems exist in the prior art:

[0003] 1. The information of engineering design drawings (paper or electronic version) is stored in a two-dimensional form, and the efficiency of reading information is low, and it is impossible to quickly extract or check design information; two-dimensional drawings cannot accurately and comprehensively describe three-dimensional designs.

[0004] 2. Non-BIM three-dimensional models and two-dimensional drawings cannot fully correspond, and cannot accurately describe three-dimensional designs; moreover, it is impossible to directly superimpose drawings and models on the same interface for comparison, and it is very brain-consuming to understand the design and prone to errors.

[0005] 3. Existing BIM engineering software tools tend to use BIM models in a pure three-dimensional environment, the information and interface are too complex and inconvenient to use. Therefore, a large number of BIM models are not fully utilized, and the actual design results used are still two-dimensional drawings, resulting in a waste of a large amount of work.

[0006] 4. On mobile devices with low hardware configuration, it is impossible to smoothly display a large number of geometric shapes and vector texts in a three-dimensional interface, and the drawings and models of complex projects must be viewed on high-end devices.

[0007] 5. High-quality three-dimensional renderings are used in the form of two-dimensional raster images, or are used with real-time rendering tools with single functions, and cannot be presented synchronously and superimposed with engineering drawings, and manual comparison is required. Summary of the Invention

[0008] The purpose of the present invention is to provide a method for using BIM data through a graphic interaction engine on a mobile device to solve at least one of the above technical problems.

[0009] The technical problems solved by the present invention can be achieved by adopting the following technical solutions:

[0010] A method for using BIM data on a mobile device through a graphics interaction engine, characterized in that, based on rules set by drawing tags and BIM, a recommended drawing list A for drawing jump links and a corresponding recommended weight vector Va are established; based on deep learning of user usage data, a recommended drawing list B for drawing jump links and a corresponding recommended weight vector Vb are generated; after the client performs clipping and normalization processing on the recommended weight vector Va and the recommended weight vector Vb, the recommended weight vectors are sorted in descending order of weight to obtain a final recommended drawing list; when the user opens a drawing, the client provides the corresponding drawing link to the user according to the final recommended drawing list. Through the optimization of jump links, the present invention establishes a link relationship between BIM 3D models, BIM 2D drawings and non-BIM 3D models, non-BIM 2D drawings, allowing them to be directly superimposed on the same interface, thereby reducing the difficulty of understanding the design.

[0011] Convert various data in the BIM model into a data format that can be transmitted over the network. Thus, it can be uploaded to the server and allowed to be downloaded by the client for use. Thereby balancing the efficiency of data transmission and improving the flexibility of data format version changes.

[0012] Split the three-dimensional shape into several small pieces and distribute them into several frames. Thereby greatly reducing the waiting and lag caused by the operation of the main thread tasks.

[0013] Apply a variety of graphics and interaction technologies to present data, including: drawing superposition, simultaneous presentation of drawings and models, and automatic filling of cross-sections. Thereby achieving a smooth, beautiful and unique graphics interaction effect on the mobile device, overcoming the deficiencies of traditional paper drawings and conventional electronic drawings, and accurately, intuitively and comprehensively expressing the information of design drawings and models. Brief Description of the Drawings

[0014] Figure 1 is a flowchart of the present invention. Detailed Embodiment

[0015] In order to make the technical means, creative features, achieved purposes and effects of the present invention easy to understand, the present invention will be further described below with reference to specific drawings.

[0016] Refer to Figure 1 , a method for using BIM data on a mobile device through a graphics interaction engine, includes the following steps:

[0017] 1. Establish a weighted directed graph (type A link) that links each drawing based on predetermined rules and is used for pre-reading and drawing jump recommendation functions. The specific operations are as follows:

[0018] a. Establish drawing tags. There are two methods for establishment:

[0019] i. Automatically read from the BIM model. The metadata of the drawing attributes in the BIM model exists in the BIM database and is directly read according to the pre-defined rules.

[0020] ii. OCR automatically recognizes the frame information to generate labels of the drawing-related attributes. Compared with directly reading information according to the rules, OCR has lower requirements for the consistency of data sources and is applied to some special drawings lacking metadata or BIM models with inconsistent drawing data rules.

[0021] b. Establish Class A links during BIM data processing: Automatically establish jump links between drawings based on drawing labels and set rules, achieving the effect of saving a large amount of work in flipping through and searching for drawings when viewing drawings.

[0022] The types of Class A links include:

[0023] i. Upper and lower floors: Floor plans of adjacent floors, such as the first-floor plan and the second-floor plan, the top-floor plan and the roof plan.

[0024] ii. Adjacent areas: Floor plans of adjacent planar areas, such as the south area and the north area of a single building. iii. Detail drawings referred from: Small-scale drawings referred from large-scale drawings, such as establishing links between the building plan and the stair detail plan, the wall section detail plan, and between the combined plan and the sub-item plan.

[0025] iv. Content derivation: Derived drawings using the same base map, such as establishing links between the master plan and the greening master plan, the fire protection master plan, etc.

[0026] v. Spatial relationship: Establish links according to the spatial relationship, such as establishing links between the wall section detail plan, the door and window detail drawing and the corresponding building elevation.

[0027] vi. Specified association: Establish drawing links according to the logical relationship, such as establishing links between the master plan and the first-floor plans of each sub-item building drawings.

[0028] c. Generate a weighted directed graph (Weighted-Directed-Graph) through Class A links, and the implementation steps are as follows:

[0029] i. Create a corresponding node (Node) for each drawing and save the drawing's Id information in the node data.

[0030] ii. Simply traverse all Class A links to create corresponding edges (Edge). The data included in each edge: the starting node and the target node, the distance of the edge, and the type of the edge (corresponding to the link type).

[0031] iii. Generate a recommended drawing list A and the corresponding recommended weight vector Va from the links.

[0032] 2. Personalized recommendation links (Type B links) generated through deep learning based on user usage data. Technical implementation:

[0033] a. After the user operation data is obtained from the client, it is uploaded to the cloud and converted into a vector through an encoder in the cloud

[0034] Xt and uploaded to the cloud.

[0035] b. The cloud recommendation system uses the LSTM model as the core, combines the user's long-term operation history Ct-1 and the current client state Xt, infers the predicted state vector Ht, and updates Ct-1 to Ct. Ct is used as one of the input data for the next inference operation.

[0036] c. The current optional recommendation items on the client are converted into vectors through an encoder, concatenated with Ht, input into an ANN model, and the final recommendation result is output. The recommendation result is the drawing list B and the corresponding recommendation weight vector Vb.

[0037] 3. When the user opens a drawing, the client combines Type A links and Type B links to provide drawing jump recommendations.

[0038] This technology can reduce a large amount of manual work of searching and flipping through drawings, and improve the efficiency and accuracy of drawing reading for engineering designers. Technical implementation:

[0039] a. The client obtains the weight vectors Va and Vb according to the above method, and reorders the dimension values from largest to smallest respectively.

[0040] b. According to the smaller one of the two vectors in terms of dimension, trim the last dimension of the other vector to make the two vectors have the same dimension.

[0041] c. Normalize the trimmed weight vectors respectively

[0042] d. Concatenate the two weight vectors, sort them from largest to smallest by weight, and obtain the final recommended drawing list from the AB list in this order, and recommend a specified number of drawing links for the user to choose.

[0043] 4. Storage and reading of data

[0044] a. Data format: Various data in the BIM model are converted into a data format that can be transmitted over the network using different serialization algorithms, uploaded to the server, and downloaded to the client for use. The reasonable application of various data formats can balance the efficiency of data transmission and the flexibility of data format version changes.

[0045] i. For 2D drawing data and metadata, the MessagePack binary serialization algorithm is used, which allows for the free definition of data structures, highly compresses the data volume, and enables high-speed storage and reading. ii. For 3D model data, the GLTF binary serialization algorithm is used, and the 3D mesh data is highly compressed, and high-speed storage and reading are achieved.

[0046] iii. For other drawings and model metadata with relatively small data volumes, the JSON serialization algorithm is used, which allows for the free definition of data structures.

[0047] b. Data pre-reading function module: Using the A-type link weighted directed graph in the above text, a pre-reading order list is generated, and data is automatically downloaded in the background while the user is operating, and the download order conforms to the user's most likely browsing order. This technology can reduce the situation where users wait for the download to complete.

[0048] Technical implementation:

[0049] i. Reset the distance value Dn of all nodes to a very large value

[0050] ii. Create an Agent, and start a depth-first traversal of the entire graph from the node corresponding to the drawing currently used by the user. The initial distance value Da of the Agent is zero.

[0051] iii. Every time the Agent crosses an edge, add the reciprocal of the edge weight to the Agent distance value Da, and assign Da to the distance value Dn of the current node

[0052] iv. After the traversal, sort all nodes in ascending order according to the distance value Dn, and download the drawing data from the server in this order.

[0053] 5. The generation of the shapes of drawings and models uses the coroutine technology. The 3D shape is generated in the main thread. Using the coroutine technology, the task of generating the shape is automatically split into several small pieces according to the specified workload per frame and distributed over several frames. A dedicated controller controls the running time of all coroutine operations in each frame, and allocates the available running time according to the importance of the coroutines. The more important the coroutine, the longer it runs. This technology can greatly reduce the waiting and lag caused by the operation of tasks in the main thread.

[0054] 6. Use of the graphics interaction engine: Apply multiple graphics and interaction technologies to present data, including: drawing overlay, simultaneous presentation of drawings and models, and automatic filling of cross-sections. These technologies can achieve smooth, beautiful and unique graphics interaction effects on mobile devices, overcome the deficiencies of traditional paper drawings and conventional electronic drawings, and accurately, intuitively and comprehensively express the information of design drawings and models.

[0055] a. Drawing Overlay: When the user views Drawing A, the content of Drawing B can be overlaid and displayed. The content of Drawing B is displayed within a fixed range, and is not displayed outside this range. This range can move together with Drawing A or be a fixed range on the screen. The display of both Drawing A and Drawing B within this range is different from the normal display. Background operations use the GPU shader, and the above effects can be achieved without generating additional geometries and materials. Technical implementation:

[0056] i. When generating the geometries of the drawings for the first time, use the shader to create a dedicated material for each drawing and initialize the drawing coordinate system attribute values of this material;

[0057] ii. When the user views Drawing A, they choose to overlay the content of Drawing B. If the data of Drawing B has not been loaded yet,

[0058] then download and generate Drawing B;

[0059] iii. Calculate the corner coordinates of the current window, offset inwards by a specified distance, convert from screen pixel coordinates to three-dimensional coordinates on the plane where Drawing B is located through the camera projection matrix, and then convert to two corner points within the plane of Drawing B through the coordinate system of Drawing B, that is, the two-dimensional rectangular range within the plane of Drawing B is used as the overlay area.

[0060] iv. The objects corresponding to Drawing B are activated. Set the overlay-related attributes in the materials of Drawings A and B:

[0061] 1. The overlay boolean value is 1

[0062] 2. The corner point attributes of the overlay area are the corner coordinates from the previous step

[0063] 3. Display settings inside and outside the overlay area

[0064] v. When the shaders of the materials of Drawings A and B calculate each pixel point, first convert the three-dimensional coordinates of the pixel point into two-dimensional coordinate points within the coordinate system of Drawing B, and then determine whether the pixel point is within the overlay rectangle area. If so, render the pixel point according to the visual settings of the overlay; if not, render the pixel point according to the non-overlay visual settings. If the visual setting is not to display, then skip the rendering of this pixel.

[0065] b. The drawings and models are presented at the same time. The models can be cut and the cut surface is automatically filled. Using Collider, Draw Call Batching, Runtime Occlusion Culling, GPU shader, and Render Texture, a large number of geometric shapes can be displayed and interacted with smoothly and with high quality on mobile devices.

[0066] i. Draw Call Batching: When generating 3D shapes, call the graphics engine's Draw Call Batching function to reorganize the scattered 3D shapes distributed in a large number of components into several merged meshes. When rendering each frame, the batched meshes are sent to the GPU for rendering.

[0067] ii. Real-time occlusion culling: When generating a 3D shape, a box collider and a mesh collider that wraps the 3D shape are generated at the same time. Before rendering each frame, a ray is emitted from the camera viewpoint to calculate whether each box collider is visible, that is, whether it is blocked by other colliders. If visible, the mesh collider is calculated to determine whether the mesh of this object needs to be rendered in this frame.

[0068] iii. Drawing range collision elimination: The visibility of the 3D model is initially invisible. Create a new box collider within the 3D range displayed in the drawing, detect all 3D models that intersect with the collider and set them to be visible.

[0069] iv. Cutting 3D model: The internal structure of the complex model needs to be presented through 3D cutting, which corresponds to the cross-section in the design drawing. Technical implementation:

[0070] 1. Create materials through custom shaders when generating 3D shapes

[0071] 2. Write the coordinate system matrix of the drawing into the shader video memory

[0072] 3. When the shader renders each pixel, it reads the paper coordinate system matrix, converts the current pixel point to world coordinates, and determines whether it is in front of the plane where the paper is located. If so, the rendering of the pixel point is skipped.

[0073] c. Automatic filling of section planes: 3D mesh shapes have no volume information, and the interior of the shape is displayed as a hole when cut. In order to meet the drawing viewing habits of engineering project personnel, a function of automatically filling the section plane is required. Technical implementation:

[0074] i. Use the drawing sectioning range collider to find the 3D shape to be sectioned

[0075] ii. Obtain each original object mesh and material from the merged meshes of the batch processing

[0076] iii. Input the mesh and material into multiple shaders for rendering. Here, there are two rendering paths:

[0077] 1. Path A

[0078] a. Set shader A: Turn off depth culling, turn on back face culling, and set the render queue to transparent

[0079] b. Input the mesh and material into shader A1.

[0080] c. Each time shader A renders a pixel, increment it by 1 in the pixel cache

[0081] d. Input the mesh and material into shader A2.

[0082] e. Shader A2 determines whether each pixel cache is odd. If it is odd, render it with the cut fill color, otherwise skip the rendering. The rendering target is render texture A

[0083] 2. Path B

[0084] a. Set shader B: Turn on depth culling, turn off back face culling, and set the render queue to transparent

[0085] b. Determine whether the pixel is a back face. If it is a back face, render it with the cut fill color, otherwise skip the rendering. The rendering target is render texture B.

[0086] iv. Repeat step 3 for each mesh.

[0087] v. Input render textures A and B into shader C to merge the pixels and obtain render texture C.

[0088] vi. Overlay render texture C on the rendering result of the current frame to achieve the cut plane filling effect.

[0089] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification only illustrates the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.

Claims

1. A method for using BIM data on a mobile device through a graphic interaction engine, characterized in that, Based on the rules set by drawing tags and BIM, establish a recommended drawing list A for generating type A drawing jump links and the corresponding recommended weight vector Va; The types of type A links include floor plans of adjacent floors and floor plans of adjacent planar regions; The method for generating the recommended drawing list A and the corresponding recommended weight vector Va is as follows: Step 1. Create corresponding nodes for each drawing and save the drawing's Id information in the node data; Step 2. Traverse all type A links to create corresponding edges, and the data included in each edge: the start node and the target node, the distance of the edge, and the link type corresponding to the edge; Step 3. Generate the recommended drawing list A and the corresponding recommended weight vector Va from the links; Based on the deep learning of user usage data, generate a recommended drawing list B for generating type B drawing jump links and the corresponding recommended weight vector Vb; Type B links are personalized recommended links generated based on deep learning of user usage data; The method for generating the recommended drawing list B and the corresponding recommended weight vector Vb is as follows: Step 1. After obtaining the user operation data from the client, upload it to the cloud, and convert it into a vector Xt through an encoder in the cloud and upload it to the cloud; Step 2. The cloud recommendation system uses the LSTM model as the core, combines the user's long-term operation history Ct-1 and the current client state Xt, infers the predicted state vector Ht, and updates Ct-1 to Ct. Ct is used as one of the input data for the next inference operation; Step 3. The current optional recommended items on the client are converted into a vector through an encoder, concatenate this vector with Ht, input it into an ANN model, and output the final recommendation result, which is the drawing list B and the corresponding recommended weight vector Vb; After the client performs clipping and normalization processing on the recommended weight vector Va and the recommended weight vector Vb, sort the recommended weight vectors in descending order of weight to obtain the final recommended drawing list; When the user opens a drawing, the client provides the corresponding drawing link to the user according to the final recommended drawing list; 2. The method for using BIM data on a mobile device through a graphics interaction engine according to claim 1, characterized in that, The method for clipping the recommended weight vector Va and the recommended weight vector Vb: Step 1. Re-sort the dimension values of the recommended weight vector Va and the recommended weight vector Vb in descending order respectively; Step 2. Clip the end dimension of the other vector according to the smaller dimension of the two vectors to make the dimensions of the two vectors equal.

3. The method for using BIM data on a mobile device through a graphic interaction engine according to claim 1, wherein Convert various data in the BIM model into a data format that can be transmitted over the network using different serialization algorithms: Among them, two-dimensional drawing data and metadata use the MessagePack binary serialization algorithm, three-dimensional model data uses the GLTF binary serialization algorithm, and other drawings and model metadata with relatively small data volumes use the JSON serialization algorithm.

4. The method of using BIM data on a mobile device through a graphics interaction engine according to claim 1, wherein Use the coroutine technology to automatically split the task of generating shapes into several small pieces according to the specified workload per frame, allocate them to several frames, and the controller controls the running time of all coroutine operations in each frame, and allocates the available running time according to the importance of the coroutines. The more important the coroutine, the longer the running time.

5. The method for using BIM data on a mobile device through a graphics interaction engine according to claim 1, wherein Apply multiple graphics and interaction technologies to present the drawing overlay. The specific steps are as follows: Step 1. When generating the geometric shapes of the drawings for the first time, use a shader to create a dedicated material for each drawing and initialize the property values of the drawing coordinate system of the material. Step 2. When the user views Drawing A and selects to overlay the content of Drawing B, if the data of Drawing B has not been loaded yet, download and generate Drawing B. Step 3. Calculate the corner coordinates of the current window, offset inward by a specified distance, convert from the screen pixel coordinates to the three-dimensional coordinates of the plane where Drawing B is located through the camera projection matrix, and then convert to two corner points within the plane of Drawing B through the coordinate system of Drawing B, that is, the two-dimensional rectangular range within the plane of Drawing B is used as the overlay area. Step 4. The object corresponding to Drawing B is activated, and the properties related to overlay in the materials of Drawings A and B are set. Step 5. When the shaders of the materials of Drawings A and B calculate each pixel point, first convert the three-dimensional coordinates of the pixel point to the coordinate system of Drawing B to become a two-dimensional coordinate point, and then determine whether the pixel point is within the overlay rectangular area: if so, render the pixel point according to the visual settings of the overlay; if not, render the pixel point according to the visual settings of non-overlay. If the visual setting is not to display, skip the rendering of this pixel.

6. The method for using BIM data on a mobile device through a graphic interaction engine according to claim 1, characterized in that Apply multiple graphics and interaction technologies to present the drawing overlay with the model simultaneously. The specific steps are as follows: Step 1. Draw call batching: When generating three-dimensional shapes, call the draw call batching function of the graphics engine to reorganize the scattered three-dimensional shapes distributed in a large number of components into several merged meshes, and send the batched meshes to the GPU for rendering during each frame of rendering. Step 2. Real-time occlusion culling: When generating three-dimensional shapes, generate a box collider and a mesh collider that enclose the three-dimensional shape at the same time. Before each frame of rendering, emit a ray from the camera viewpoint and calculate whether each box collider is visible, that is, whether it is occluded by other colliders. If it is visible, then calculate whether the mesh collider is visible to determine whether the mesh of this object needs to be rendered in this frame. Step 3. Drawing range occlusion culling: The visibility of the three-dimensional model is initially invisible. Create a new box collider within the three-dimensional range where the drawing is displayed, and detect all three-dimensional models that intersect with this collider and set them to be visible.

7. The method for using BIM data on a mobile device through a graphic interaction engine according to claim 1, characterized in that Apply multiple graphics and interaction technologies for automatic filling of cutting planes. The specific steps are as follows: Step 1. Use the drawing cutting range collider to find the three-dimensional shapes that are cut and displayed. Step 2. Obtain each original object mesh and material from the merged meshes processed by batching. Step 3. Input the mesh and material into multiple shaders for rendering: Set shader A: Turn off depth culling, turn on backface culling, and set the rendering queue to transparent; Input the mesh and material into shader A1; Each time shader A renders a pixel, increment by 1 in the pixel buffer; Input the mesh and material into shader A2; Shader A2 determines whether each pixel buffer is odd. If it is odd, render it according to the cutting filling color, otherwise skip the rendering. The rendering target is rendering texture A; Set shader B: Turn on depth culling, turn off backface culling, and set the rendering queue to transparent. Determine whether the pixel is the back side. If it is the back side, render it according to the cut fill color. Otherwise, skip rendering and the rendering target is rendering map B. Step 4. Repeat step 3 for each mesh; Step 5: Input the rendering maps A and B into shader C, merge the pixels, and obtain the rendering map C; Step 6: Overlay the rendering map C on the rendering result of the current frame to achieve the section surface filling effect.

Images