Multimodal digital twin scene construction method and device based on image semantic fusion

Abstract

The multimodal digital twin scene construction method and device based on image semantic fusion provided by the present invention relate to the technical field of data processing and digital twin scene. The present invention obtains image data sets and text data sets of a construction site and performs preprocessing; inputs the preprocessed image data sets and text data sets into a natural language processing model to obtain semantic description auxiliary data; inputs the preprocessed image data sets into a target detection algorithm to obtain target extraction image data containing a target detection frame; then fuses the output results of the two to obtain a fusion frame of each target; then uses a perspective projection algorithm to calculate the coordinate position of each target corresponding to the BIM to obtain a three-dimensional world coordinate; uses Dynamo to input the three-dimensional world coordinate into the BIM to realize the construction and real-time update of a multimodal digital twin scene. The present invention significantly improves the accuracy and robustness of scene recognition and solves the shortcomings of traditional methods in multimodal data processing.

Description

Technical Field

[0001] The present invention relates to the field of data processing and digital twin scene technology, and in particular, to a multimodal digital twin scene construction method and device based on image semantic fusion. Background Art

[0002] With the rapid development of information technology, the construction industry has improved construction efficiency and safety by integrating advanced artificial intelligence technology. However, safety management of construction sites still faces many challenges, including the identification and assessment of potential hazards in complex environments. In addition, there is still a certain disconnect between the existing building information model (BIM) and the real-time monitoring and management of the construction site, resulting in the inability to promptly resolve safety and efficiency issues that may arise during the construction process.

[0003] Existing technologies can obtain a large amount of real-time data through information technology, but how to accurately extract and improve safety management benefits from these data. In addition, the heterogeneity and complexity of data and the mismatch between the building information model (BIM) and the site make information integration and analysis have multiple interfaces and insufficient immediacy. How to improve the integration of construction safety management under the dynamic and unpredictable working environment conditions of the construction site, and the interaction between various elements such as personnel, materials, machinery, etc., and the construction site and environment, faces the following challenges:

[0004] ① It is difficult to collect construction site data and the data quality is uneven. There are still certain difficulties in accurately extracting the required information from these data and improving the safety management benefits, which affects the accuracy and reliability of the prediction.

[0005] ② The data models and standards of different sites are not unified, and there is a lack of a standardized way to describe the status of the construction site, making it difficult to achieve effective data sharing and integration.

[0006] ③ When dealing with dynamic changes and complex scenarios, the real-time performance is insufficient, and there is a mismatch between the BIM model and the site, which makes it difficult to meet the needs of rapid response at the construction site.

[0007] In view of this, the applicant filed this application after studying the existing technology. Summary of the invention

[0008] The present invention aims to provide a multimodal digital twin scene construction method and device based on image semantic fusion, so as to solve the problems existing in the existing methods such as the difficulty in collecting and processing multimodal data and inaccurate description of construction scenes.

[0009] In order to solve the above technical problems, the present invention is implemented through the following technical solutions:

[0010] A multimodal digital twin scene construction method based on image semantic fusion, comprising:

[0011] S1, obtain the image dataset and text dataset of the construction site and perform preprocessing;

[0012] S2, input the preprocessed image dataset and text dataset into the natural language processing model for text semantic description, and output the logical and structured text sentences corresponding to each image, so as to obtain semantic description auxiliary data;

[0013] S3, input the preprocessed image data set into the target detection algorithm for target classification detection, and output each target object image selected by the classification box, so as to obtain the target extraction image data containing the target detection box;

[0014] S4, fusing the target extraction image data and the semantic description auxiliary data to obtain a fusion frame of each target;

[0015] S5, according to the fusion frame, using a perspective projection algorithm, calculate the coordinate position information of each target in the pre-built BIM three-dimensional space, obtain the three-dimensional world coordinates of the target, and map the two-dimensional coordinate information of each target to the three-dimensional space;

[0016] S6, uses the Dynamo visual programming language to input the 3D world coordinates of each target into the pre-built BIM model to achieve the construction and real-time update of the multimodal digital twin scene.

[0017] Preferably, the image data set is obtained by photographing images of the construction site using a camera or a sensor; the text data set is constructed based on building-related regulations, construction records, safety instructions, and detailed text description information corresponding to the image data set.

[0018] Preferably, the pretreatment comprises:

[0019] Performing cropping, resizing and normalization resetting operations on the image data of the image data set;

[0020] Based on the GPT model, dangerous words are extracted and reorganized on the text data of the text data set.

[0021] Preferably, the natural language processing model is an improved natural language processing model that introduces the Transformer attention mechanism, specifically as follows:

[0022] An improved image encoder is used, that is, the image encoder of the natural language processing model is replaced with an image encoder based on Transformer; the input image data is divided into multiple image blocks of fixed size, and after passing through the linear embedding layer, it is sent to the Transformer encoder to extract the target objects existing in the corresponding scene in the image data;

[0023] An improved text encoder is used, that is, the text encoder of the natural language processing model is replaced with a Transformer-based text encoder; the input text data is converted into word vectors by the embedding layer and then sent to the Transformer encoder to match the content of the image data with the text data, and output the logical and structured text sentences corresponding to each image.

[0024] Preferably, the target detection algorithm is an improved target detection model that introduces the Transformer attention mechanism, which is as follows:

[0025] Adopt an improved feature extractor, that is, introduce the Transformer module into the feature extractor of the object detection model, and use position encoding to maintain spatial information;

[0026] Adopt an improved cross-layer attention module, that is, design a cross-layer attention module to allow interaction and information transfer between features of different layers;

[0027] An improved target detection head is adopted. That is, in the target detection head, the Transformer attention mechanism is introduced to enhance the focus on the detection target; and the attention weights are used to dynamically weight different feature maps to improve the detection accuracy.

[0028] Preferably, fusing the target extraction image data with the semantic description auxiliary data specifically comprises: according to the target extraction image data and the semantic description auxiliary data, fusing the target frame of the target extraction image data with the text frame of the semantic description auxiliary data to obtain a more accurate fusion frame of the target, the steps are as follows:

[0029] Step 1: Define any target box , , , , and the corresponding text box , , , ;

[0030] in, , Target box The coordinates of the upper left corner, , Target box The coordinates of the lower right corner; , For text box The coordinates of the upper left corner, , For text box The coordinates of the lower right corner;

[0031] Step 2: Calculate the target box With text box The coordinates of the overlapping parts , , , , the coordinate values ​​are:

[0032] =max( , ); ;

[0033] min( , ); ;

[0034] like or , then the target box With text box No overlap;

[0035] Otherwise, if they overlap, calculate the overlapping area for:

[0036] ( );

[0037] Step 3: Define a fusion box F , , , , the fusion box F is the weighted average position of the area of ​​the target box A and the area of ​​the text box B:

[0038] =frac( + )( + );

[0039] ( + );

[0040] )( + );

[0041] ( + );

[0042] in, (.) represents a fraction, which is the Fraction class of the Python language, used to accurately represent and process fractions; Represents the area of ​​the target box A; Represents the area of ​​text box B.

[0043] Preferably, based on Python language, according to the calculated fusion frame of each target, a perspective projection algorithm is used to map it to three-dimensional space, and the 3D space coordinate position information of each target is calculated, specifically:

[0044] The angle between the camera and the two edges of the object passing through the maximum range of the lens is called the field of view FOV. If the object exceeds the range of the field of view FOV, it will not be captured by the camera. The angle of the field of view is , i.e. the diagonal field of view;

[0045] According to the diameter of the imaged object on the diagonal of the rectangular photosensitive surface, the diagonal field of view is used. Calculate the object distance s. Since the field of view of the camera is fixed and known, and the range that can be captured is also related to the focal length of the lens, the formula for the object distance s that the camera can capture is:

[0046] ;

[0047] in, is the diameter of the imaged object within the visible range of the lens, c is the point of the target image, and is also the intersection of the optical axis and the reference plane;

[0048] The two-dimensional image captured by the camera is a projection of the three-dimensional space on a plane. Based on the camera posture and the coordinates of the target object in the image, that is, the fusion frame of the target, the position of the target object in the real world is obtained;

[0049] Assume that the three-dimensional coordinates of the camera are point H (0, h, 0), and the pitch angle between H and the horizontal direction is θ; the reference plane is the plane where the target object is located, and is set on the same plane as the center of the target object. The straight line s is the object distance that the camera can capture, that is, the optical axis, and the point c ( , 0, 0) is the point of the target image, which is also the intersection of the optical axis and the reference plane. The coordinates of point c and the trigonometric function relationship of angle θ are obtained as follows:

[0050] ;

[0051] ;

[0052] According to the perspective projection principle, the image captured by the camera is perpendicular to the optical axis, and the projection on the reference plane is a trapezoid, so the midpoint of the target image is set to point c; then, for any two-dimensional coordinate point Q in the target image ( , ), whose value is obtained by measuring transformation, and the corresponding three-dimensional coordinates of Q in the three-dimensional coordinate system are calculated as follows: , the formula is:

[0053] ;

[0054] So as to obtain the passing point The equation of the straight line l between and H is:

[0055] ;

[0056] Then the coordinates of the intersection point P of the straight line l and the reference plane are ( , , ), the three-dimensional coordinate formula of point P is:

[0057] ;

[0058] Thus, the two-dimensional coordinate point Q is mapped to the corresponding world coordinate point P in the digital twin scene;

[0059] The two-dimensional coordinates of each target in the image are converted into three-dimensional coordinates; then the three-dimensional coordinates of the target object are transformed by the perspective projection of the camera to obtain the world coordinates of all targets mapped to the digital twin scene.

[0060] Preferably, it also includes: using an average error distance loss function to optimize the error of the perspective projection algorithm, according to the world coordinate position P of the projected target object captured by the set camera and the three-dimensional coordinates of the target object The measurement accuracy between them is calculated, and the Euclidean distance is used to calculate the mean error distance loss MED. By calculating the Euclidean distance of each target object, the accuracy of the perspective projection algorithm is obtained. The specific calculation formula is:

[0061] ;

[0062] ;

[0063] in: represents the Euclidean distance of the i-th target; N represents the total number of samples; MED value and The smaller the value, the higher the accuracy of the perspective projection algorithm.

[0064] Preferably, the improved natural language processing model and the improved target detection algorithm are optimized and learned using a contrast loss function to fine-tune the model parameters.

[0065] The present invention also provides a multimodal digital twin scene construction device based on image semantic fusion, comprising:

[0066] Data acquisition and preprocessing module, used to obtain image data sets and text data sets of construction sites and perform preprocessing;

[0067] The semantic description module is used to input the preprocessed image dataset and text dataset into the natural language processing model for text semantic description, and output the logical and structured text sentences corresponding to each image, so as to obtain semantic description auxiliary data;

[0068] The target extraction module is used to input the preprocessed image data set into the target detection algorithm for target classification detection, and output each target object image selected by the classification box, so as to obtain the target extraction image data containing the target detection box;

[0069] An image semantic fusion module, used to fuse the target extraction image data with the semantic description auxiliary data to obtain a fusion frame of each target;

[0070] A perspective mapping module is used to calculate the coordinate position information of each target in the pre-built BIM three-dimensional space according to the fusion frame by using a perspective projection algorithm, and obtain the three-dimensional world coordinates of the target, so as to map the two-dimensional coordinate information of each target into the three-dimensional space;

[0071] The dynamic digital twin scene module is used to input the three-dimensional world coordinates of each target into the pre-built BIM model using the Dynamo visual programming language, thereby realizing the construction and real-time update of multimodal digital twin scenes.

[0072] The present invention also provides a multimodal digital twin scene construction device based on image semantic fusion, including a processor and a memory, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a multimodal digital twin scene construction method based on image semantic fusion as described above.

[0073] The present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, a multimodal digital twin scene construction method based on image semantic fusion is implemented as described above.

[0074] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0075] The present invention utilizes the image-semantic comparison technology and target detection algorithm of the natural language processing model, combined with the extraction and description of multimodal data, to propose a multimodal digital twin construction scene construction method based on image-semantic fusion, the core of which is to deeply combine the image data of the construction site with the text information, and to comprehensively and comprehensively describe the construction scene. At the same time, through the perspective mapping principle and the Dynamo visual programming language, a comprehensive, accurate and real-time digital twin construction scene is constructed, and then a set of standardized SOP operating procedures are formed to achieve efficient management and accurate decision-making of the construction site.

[0076] The present invention aims to address the problems of image data annotation at construction sites, as well as the challenges of non-structural, format diversity, and inconsistent quality of construction text data. In addition, the present invention uses advanced machine learning and deep learning technologies to systematically achieve efficient collection and processing of large amounts of heterogeneous data, targeting the small size and high impurity content of IoT sensors during data collection. The method of the present invention can significantly improve the quality of image and text data, simplify the annotation process, and enhance the ability to integrate data, thereby providing more reliable data support for the construction field.

[0077] The method of the present invention integrates the natural language processing model and the target detection model that introduce the Transformer attention mechanism to perform image-language fusion. The natural language processing model is used to perform semantic description of the scene, and the target detection model is used to perform scene recognition and classification, and then the two are integrated to achieve a comprehensive description of the architectural scene. The method of the present invention significantly improves the accuracy and robustness of scene recognition, and solves the shortcomings of traditional methods in multimodal data description and processing.

[0078] The present invention uses the perspective mapping principle to effectively convert the two-dimensional coordinate information of the image into 3D space coordinate information, and uses the Dynamo visual programming language to map these coordinates to the BIM model, and dynamically adjusts the model in real time to match the actual situation on the construction site, thereby effectively solving the problem of data update lag in traditional methods, improving the accuracy and timeliness of the model, and enhancing the dynamic response capability of the building information model. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0080] Figure 1A schematic diagram of a multimodal digital twin scene construction method based on image semantic fusion provided in Example 1.

[0081] Figure 2 A flowchart of a multimodal digital twin scene construction method based on image semantic fusion provided in Example 1.

[0082] Figure 3 This is a flowchart of image data processing provided in Example 1.

[0083] Figure 4 A schematic diagram of text data processing provided in Example 1.

[0084] Figure 5 A schematic diagram of the image-text information fusion calculation provided in Example 1.

[0085] Figure 6 This is a schematic diagram of the field of view angle model provided in Example 1.

[0086] Figure 7 This is a schematic diagram of the diagonal field of view model provided in Example 1.

[0087] Figure 8 This is a schematic diagram of the perspective projection principle model provided in Example 1.

[0088] Fig. 9 A schematic diagram of a multimodal digital twin scene construction device based on image semantic fusion provided in Example 2.

[0089] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0090] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0091] Embodiment 1

[0092] Embodiment 1 of the present invention provides a multimodal digital twin scene construction method based on image semantic fusion, which can be implemented by a multimodal digital twin scene construction device based on image semantic fusion (hereinafter referred to as the digital twin scene construction device), and in particular, executed by one or more processors in the digital twin scene construction device.

[0093] In this embodiment, the digital twin scene construction device may be an electronic device equipped with a processor, which carries a computer program of the multimodal digital twin scene construction method based on image semantic fusion and the computer program can be executed, such as a computer, a smart phone, a smart tablet, a workstation, etc., which is not limited here.

[0094] In this embodiment, BIM (Building Information Modeling) is used to describe computer-aided designs that are mainly three-dimensional graphics, object-oriented, and related to architecture.

[0095] Revit: A BIM modeling software developed by Autodesk.

[0096] CLIP model: A Contrastive Language-Image Pre-training method, released by OpenAI in 2021, is a multimodal model that combines computer vision and natural language processing techniques.

[0097] YOLO algorithm: A real-time target detection algorithm based on convolutional neural networks (You Only Look Once), which can simultaneously locate and classify targets through a single neural network to achieve real-time and efficient target detection.

[0098] Dynamo: A visual programming tool that allows you to intuitively view script behavior, define custom logic, and use scripts in various text programming languages. It can be used with Revit software.

[0099] Perspective Projection: Also known as perspective drawing, it is a method of drawing or rendering on a two-dimensional paper or canvas plane in order to obtain a visual effect close to a real three-dimensional object.

[0100] Field of View (FOV): In optical instruments such as cameras, the angle formed by the two edges of the maximum range through which the image of the target can pass through the lens, with the lens of the optical instrument as the vertex, is called the field of view.

[0101] Euclidean Metric: Also known as Euclidean distance, it is a commonly used distance definition, which refers to the true distance between two points in n-dimensional space, or the natural length of a vector (that is, the distance from the point to the origin). The Euclidean distance in two-dimensional and three-dimensional space is the actual distance between two points.

[0102] Mean Error Distance (MED): A measure of the average error of the predictions.

[0103] SOP: Standard Operating Procedure, refers to the standard operating procedures and requirements of an event described in a unified format to guide and standardize daily work.

[0104] The existing digital twin scene construction technology solutions are mainly based on the following methods:

[0105] (1) Use cameras, mobile devices and IoT sensors to collect images, texts and various physical quantities at the construction site, and transmit the data in real time through the network.

[0106] However, the existing technology is time-consuming and laborious for high-quality annotation of image data, and the accuracy of annotation directly affects the performance of the model. For text data, the text data in the construction scene may come from different sources and in different formats. The content covers engineering logs, material lists, safety reports, etc., and most of them are unstructured text. In addition, the site needs to obtain text data in real time to make quick decisions, but the traditional text processing process is often time-consuming, and it is also difficult to extract the required content from a large amount of text for analysis. For IoT sensors, their operation depends on a stable network connection. If the network environment is not good, it may affect the transmission and real-time performance of the data. In addition, the data collected by IoT sensors is difficult to graft into the BIM model. Professional technicians are required to connect the network. With the increase of equipment, the cost and technology of maintenance and upgrade may increase.

[0107] (2) Use the GNN model to analyze construction site pictures and extract relevant information; use computer vision technologies such as the YOLO model to perform object detection on construction site pictures.

[0108] Although GNN can directly process image structure data and has good modeling capabilities for the relationship between nodes and topological structures, it still lacks a simple and accurate description mechanism for describing the elements in the image and the relationship between them. In addition, GNN is limited to processing image data and cannot process text data. Although the YOLO series is constantly being optimized to improve detection speed and accuracy, its main disadvantage is that the detection accuracy of small targets and overlapping objects still needs to be improved.

[0109] (3) Use Revit software for BIM modeling to achieve digital expression of building information.

[0110] The BIM model uses Revit software to import and model the CAD drawings of each construction stage of the construction project, which will cause the BIM model to be inconsistent with the real-time data of the construction site, making it difficult to achieve real-time supervision of the construction site. For example, design changes are updated in the BIM model, but the construction site fails to obtain and implement these changes in a timely manner; workers carry out construction according to the BIM model, which may lead to material waste, rework and other problems during the construction process; managers find it difficult to accurately monitor the progress of the construction and cannot adjust the plan in time to cope with the actual situation on site.

[0111] Therefore, if Figure 1-Figure 2 As shown, a multimodal digital twin scene construction method based on image semantic fusion includes steps S1 to S6.

[0112] S1, obtain the image dataset and text dataset of the construction site and perform preprocessing.

[0113] First, a variety of sensors and equipment, such as high-definition cameras and sensors, are used to collect a large number of construction site images as well as relevant building regulations, construction records, safety instructions, etc. as image datasets and text datasets for model training.

[0114] Specifically, Figure 3 As shown in the figure, the collected image data are organized into a unified structure, and then center cropped and resized to the required image size, and then normalized. Each channel of each image is normalized, and each pixel on each channel will subtract the mean of the pixel value of the channel and divide it by the variance of the pixel value of the channel; finally, each image is converted into a multidimensional data.

[0115] like Figure 4 As shown in the figure, after the text data is refined, such as word segmentation, stop word removal and semantic annotation, the ChatGPT model is used to perform manual assisted preprocessing on the collected text data, such as extracting and reorganizing dangerous words from the text data, etc. Each image corresponds to a piece of text data.

[0116] The preprocessed image dataset and text dataset are randomly divided into training, verification and test datasets, accounting for 60%, 30% and 10% of the total dataset respectively, and the model is trained, verified and tested.

[0117] S2, input the preprocessed image dataset and text dataset into the natural language processing model for text semantic description, and output the logical and structured text sentences corresponding to each image, so as to obtain semantic description auxiliary data.

[0118] In this embodiment, the natural language processing model is an improved natural language processing model that introduces the Transformer attention mechanism, which is as follows:

[0119] An improved image encoder is used, that is, the image encoder of the natural language processing model is replaced with an image encoder based on Transformer; the input image data is divided into multiple image blocks of fixed size, and after passing through the linear embedding layer, it is sent to the Transformer encoder to extract the target objects existing in the corresponding scene in the image data;

[0120] An improved text encoder is used, that is, the text encoder of the natural language processing model is replaced with a Transformer-based text encoder; the input text data is converted into word vectors by the embedding layer and then sent to the Transformer encoder to match the content of the image data with the text data, and output the logical and structured text sentences corresponding to each image.

[0121] Finally, the improved natural language processing model is optimized and learned using the contrast loss function (of course, other existing loss functions can also be used, which is not limited here) to fine-tune the model parameters.

[0122] In this embodiment, the CLIP model is selected as the natural language processing model. Of course, other advanced natural language processing models, such as BERT or SAM models, can also be used to perform text extraction and description. These models have powerful capabilities in understanding text content and context and can provide accurate and detailed descriptions.

[0123] The preprocessed image-text dataset is input into the CLIP model to obtain the corresponding output of the model. Specifically, the image encoder of the CLIP model is used to extract the target objects in the scenes such as employees and piles of wood from the image data, and then the content of the image data is matched with the text data according to the text encoder. The final output result is: each image is equipped with a structured and logical description sentence.

[0124] The preprocessed image-text training data is input into the CLIP model for training, the model parameters are adjusted to optimize the recognition effect, and the CLIP model is tested for model accuracy, recall rate and mean average precision (mAP).

[0125] S3, input the preprocessed image data set into the target detection algorithm for target classification detection, and output each target object image selected by the classification box, so as to obtain the target extraction image data containing the target detection box;

[0126] Specifically, the target detection algorithm is an improved target detection model that introduces the Transformer attention mechanism, as follows:

[0127] Adopt an improved feature extractor, that is, introduce the Transformer module into the feature extractor of the object detection model, and use position encoding to maintain spatial information;

[0128] Adopt an improved cross-layer attention module, that is, design a cross-layer attention module to allow interaction and information transfer between features of different layers;

[0129] An improved target detection head is adopted. That is, in the target detection head, the Transformer attention mechanism is introduced to enhance the focus on the detection target; and the attention weights are used to dynamically weight different feature maps to improve the detection accuracy.

[0130] For the improved target detection algorithm, a contrast loss function is used (of course, other existing loss functions can also be used, which is not limited here) for optimization learning to fine-tune the model parameters.

[0131] In this embodiment, the YOLOv8 model is selected for target detection. The YOLO algorithm performs well in target detection, but other target detection algorithms can also be used, such as SSD, Faster R-CNN or CenterNet. These algorithms have their own advantages in accuracy and speed, and the most suitable algorithm can be selected according to specific needs.

[0132] The preprocessed image dataset is used as the input of the YOLOv8 model. With the help of YOLOv8's target detection and classification functions, the construction site images are detected, framed, and classified for workers, dangerous objects, and other targets. The final output result is that the target objects in each image are classified and framed.

[0133] The preprocessed image training data is input into the YOLOv8 model for training, the model parameters are adjusted to optimize the recognition effect, and the YOLOv8 model is tested for model accuracy, recall rate and mean average precision (mAP).

[0134] S4, fusing the target extraction image data with the semantic description auxiliary data to obtain a fusion box of each target.

[0135] Specifically, data fusion is performed according to the text sentence output in step S2 and the image coordinate information output in step S3.

[0136] According to the target extraction image data and the semantic description auxiliary data, the target frame of the target extraction image data and the text frame of the semantic description auxiliary data are fused to obtain a more accurate fusion frame of the target, and the steps are as follows:

[0137] Step 1: If Figure 5 As shown, define any target box , , , , and the corresponding text box , , , ;

[0138] in, , Target box The coordinates of the upper left corner, , Target box The coordinates of the lower right corner; , For text box The coordinates of the upper left corner, , For text box The coordinates of the lower right corner;

[0139] Step 2: Calculate the target box With text box The coordinates of the overlapping parts , , , , the coordinate values ​​are:

[0140] =max( , ); ;

[0141] min( , ); ;

[0142] like or , then the target box With text box No overlap;

[0143] Otherwise, if they overlap, calculate the overlapping area for:

[0144] ( );

[0145] Step 3: Define a fusion box F , , , , the fusion box F is the weighted average position of the area of ​​the target box A and the area of ​​the text box B:

[0146] =frac( + )( + );

[0147] ( + );

[0148] )( + );

[0149] ( + );

[0150] in, (.) represents a fraction, which is the Fraction class of the Python language, used to accurately represent and process fractions; Represents the area of ​​the target box A; Represents the area of ​​text box B.

[0151] S5, according to the fusion frame, using the perspective projection algorithm, calculate the coordinate position information of each target in the pre-built BIM three-dimensional space, obtain the three-dimensional world coordinates of the target, and map the two-dimensional coordinate information of each target to the three-dimensional space.

[0152] Specifically, based on Python language, according to the calculated fusion frame of each target, the perspective projection algorithm is used to map it to the three-dimensional space, and the 3D space coordinate position information of each target is calculated, specifically:

[0153] like Figure 6 As shown in the figure, the angle formed by the camera as the vertex and the two edges of the target object through the maximum range of the lens is called the field of view FOV; if the target object exceeds the range of the field of view FOV, it will not be captured by the camera. The angle of the field of view is , that is, the diagonal field of view.

[0154] The field of view angle includes the horizontal field of view angle (HFOV, ), vertical field of view (VFOV, ) and diagonal field of view (DFOV, ),in, , , are the angles corresponding to the field of view.

[0155] Usually, the default FOV is the horizontal field of view when there is no special description. However, for optical devices such as cameras and video cameras, since their photosensitive surfaces are rectangular, the method of the present invention calculates the field of view using the diameter of the imaged object on the diagonal of the rectangular photosensitive surface, that is, using the diagonal field of view for calculation.

[0156] like Figure 7 As shown, according to the diameter of the imaged object on the diagonal of the rectangular photosensitive surface, the diagonal field of view is used. Calculate the object distance s. Since the field of view of the camera is fixed and known, and the range that can be captured is also related to the focal length of the lens, the formula for the object distance s that the camera can capture is:

[0157] ;

[0158] in, is the diameter of the imaged object within the visible range of the lens, a and b are the two end points of the diameter, and c is the center point of the target image, which is also the intersection of the optical axis and the reference plane. is the diagonal field of view, and s is the object distance that the camera can capture.

[0159] Since the two-dimensional image captured by the camera is a projection of the three-dimensional space on a plane, the position of the target object in the real world can be determined based on the camera posture and the coordinates of the target object in the image, that is, the fusion frame of the target.

[0160] like Figure 8 As shown, let the three-dimensional coordinates of the camera be point H (0, h, 0), and the pitch angle between H and the horizontal direction be θ; the reference plane is the plane where the target object is located, and is set on the same plane as the center of the target object, that is, Figure 8 The plane formed by the X-axis and the Z-axis (y=0). Figure 7 It can be seen that the straight line s is the distance of the object that the camera can capture, that is, the optical axis, and point c ( , 0, 0) is the center point of the target image, which is also the intersection of the optical axis and the reference plane. The coordinates of point c and the trigonometric function relationship of angle θ are obtained as follows:

[0161] ;

[0162] .

[0163] in, is the value of point c on the X-axis, It is the value on the Y axis in the camera's three-dimensional coordinates.

[0164] Furthermore, according to the perspective projection principle, the image captured by the camera is perpendicular to the optical axis, and the projection on the reference plane is a trapezoid. Therefore, in order to simplify the calculation, the midpoint of the target image is set to point c. Then, for any two-dimensional coordinate point Q in the target image ( , ), whose value can be obtained by measuring the transformation, and the three-dimensional coordinates corresponding to Q in the three-dimensional coordinate system are calculated as follows: ( , , ), the formula is:

[0165] ;

[0166] So as to obtain the passing point The equation of the straight line l between and H is:

[0167] ;

[0168] Then the coordinates of the intersection point of the straight line l and the reference plane are P ( , , ), the three-dimensional coordinate formula of point P is:

[0169] ;

[0170] Thus, the two-dimensional coordinate point Q is mapped to the corresponding world coordinate point P in the digital twin scene;

[0171] The two-dimensional coordinates of each target in the image are converted into three-dimensional coordinates; then the three-dimensional coordinates of the target object are transformed by the perspective projection of the camera to obtain the world coordinates of all targets mapped to the digital twin scene.

[0172] In addition, in another preferred embodiment, in order to evaluate the accuracy of the perspective projection algorithm, the present invention introduces the mean error distance (MED) for measurement. However, since both the estimated data and the ground truth data are in the form of coordinates, the accuracy of the perspective projection algorithm is obtained by calculating the Euclidean distance of each target object.

[0173] Specifically, according to the projected target world coordinate position P captured by the set camera and the three-dimensional coordinates of the target The measurement accuracy between them is calculated, and the Euclidean distance is used to calculate the mean error distance loss MED. The specific calculation formula is:

[0174] ;

[0175] ;

[0176] in: represents the Euclidean distance of the i-th target; N represents the total number of samples; MED value and The smaller the value, the higher the accuracy of the perspective projection algorithm.

[0177] The perspective projection principle method proposed in the present invention is applicable to all factors in a construction scene, and can calculate the coordinate position of a target object when its coordinates and height are unknown, so as to carry out the next step of implementation.

[0178] In this embodiment, in addition to using the perspective principle to calculate coordinates, it is also possible to consider using the camera calibration technology in computer vision, combined with the calculation of the projection matrix to obtain the precise coordinates of the target object. In addition, using a machine learning regression model to predict the position of the target object is also a feasible alternative method.

[0179] S6, uses the Dynamo visual programming language to input the 3D world coordinates of each target into the pre-built BIM model to achieve the construction and real-time update of the multimodal digital twin scene.

[0180] In this embodiment, Dynamo is a powerful visual programming tool, but other automation tools compatible with Revit, such as Grasshopper or Rhino's Python script, can also be considered to import the target's 3D world coordinate data and implement interaction with the BIM model.

[0181] In addition to using Dynamo to import the target's three-dimensional world coordinate data into Revit to achieve digital twins, you can of course also consider using cloud computing and big data technologies to achieve dynamic updating and monitoring of BIM models by establishing a real-time data stream and model update mechanism.

[0182] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0183] The embodiment of the present invention creatively integrates CLIP model, YOLOv8 model and Dynamo real-time visualization through the extraction and description of multimodal data, and applies it to the dynamic construction scene. It constructs a construction method of dynamic construction scene based on digital twins and forms a set of standardized SOP operation procedures, which realizes efficient management and accurate decision-making of the construction site. The beneficial effects brought by each key control node are as follows:

[0184] (1) The present invention effectively solves problems such as construction site monitoring, compliance inspection and risk assessment by collecting multimodal data (such as images, text, etc.) at the construction site, thereby improving the accuracy and efficiency of construction management.

[0185] (2) The present invention can effectively improve data quality and security and enhance the accuracy of anomaly detection at construction sites by cropping, resizing and normalizing image data, and applying the GPT model to text data to extract and reorganize dangerous words, thereby timely discovering and handling potential safety hazards.

[0186] (3) The present invention can realize intelligent analysis of the construction site by inputting the preprocessed image-text dataset into the CLIP model with the Transformer attention mechanism, and inputting the image dataset into the YOLOv8 model with the Transformer attention mechanism, integrating visual and semantic information, and effectively improving the accuracy of object detection and scene understanding, thereby assisting safety monitoring and management decision-making at the construction site.

[0187] (4) The present invention fuses the image-text information output by the CLIP model and the YOLOv8 model, and applies a perspective projection algorithm to convert the image coordinate information output by the YOLOv8 model into 3D space coordinates. This process can achieve accurate positioning and quantitative analysis of target objects at the construction site, facilitate subsequent three-dimensional space management, collision detection, and construction progress monitoring, thereby improving the safety and efficiency of the construction site.

[0188] (5) By pre-building the BIM model of the detected and extracted target object in Revit, the present invention can achieve seamless connection between the construction site and the real model, facilitate comparative analysis of the virtual and real environments, thereby optimizing the construction process, improving the consistency of design and construction, and enhancing the visualization and coordination of project management.

[0189] (6) The present invention uses the Dynamo visual programming language to input the actual coordinate data of the site into the BIM model and match it with the target object model for display, which can realize real-time mapping and synchronous updating between the construction site and the virtual model, thereby improving the coordination and efficiency of construction, ensuring the consistency between the design intent and the on-site implementation, and realizing the digital twin of the dynamic construction scene.

[0190] Embodiment 2

[0191] like Fig. 9 As shown, the second embodiment of the present invention further provides a multimodal digital twin scene construction device based on image semantic fusion, comprising:

[0192] Data acquisition and preprocessing module, used to obtain image data sets and text data sets of construction sites and perform preprocessing;

[0193] The semantic description module is used to input the preprocessed image dataset and text dataset into the natural language processing model for text semantic description, and output the logical and structured text sentences corresponding to each image, so as to obtain semantic description auxiliary data;

[0194] The target extraction module is used to input the preprocessed image data set into the target detection algorithm for target classification detection, and output each target object image selected by the classification box, so as to obtain the target extraction image data containing the target detection box;

[0195] An image semantic fusion module, used to fuse the target extraction image data with the semantic description auxiliary data to obtain a fusion frame of each target;

[0196] A perspective mapping module is used to calculate the coordinate position information of each target in the pre-built BIM three-dimensional space according to the fusion frame by using a perspective projection algorithm, and obtain the three-dimensional world coordinates of the target, so as to map the two-dimensional coordinate information of each target into the three-dimensional space;

[0197] The dynamic digital twin scene module is used to input the three-dimensional world coordinates of each target into the pre-built BIM model using the Dynamo visual programming language, thereby realizing the construction and real-time update of multimodal digital twin scenes.

[0198] Embodiment 3

[0199] The third embodiment of the present invention also provides a multimodal digital twin scene construction device based on image semantic fusion, which includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the multimodal digital twin scene construction method based on image semantic fusion as described above.

[0200] Embodiment 4

[0201] The fourth embodiment of the present invention also provides a computer-readable storage medium, which stores computer-readable instructions. When the computer-readable instructions are executed by the processor of the device where the computer-readable storage medium is located, the multimodal digital twin scene construction method based on image semantic fusion as described above is implemented.

[0202] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0203] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0204] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code. It should be noted that in this article, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0205] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0206] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0207] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0208] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0209] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multimodal digital twin scene construction method based on image semantic fusion, characterized in that: include: S1, obtain the image dataset and text dataset of the construction site and perform preprocessing; S2, input the preprocessed image dataset and text dataset into the natural language processing model for text semantic description, and output the logical and structured text sentences corresponding to each image, so as to obtain semantic description auxiliary data; wherein, the natural language processing model is an improved natural language processing model that introduces the Transformer attention mechanism, which is as follows: An improved image encoder is used, that is, the image encoder of the natural language processing model is replaced with an image encoder based on Transformer; the input image data is divided into multiple image blocks of fixed size, and after passing through the linear embedding layer, it is sent to the Transformer encoder to extract the target objects existing in the corresponding scene in the image data; An improved text encoder is used, that is, the text encoder of the natural language processing model is replaced with a text encoder based on Transformer. The input text data is converted into word vectors by the embedding layer and then sent to the Transformer encoder to match the content of the image data with the text data and output the logical and structured text sentences corresponding to each image. S3, input the preprocessed image data set into the target detection algorithm for target classification detection, and output each target object image selected by the classification box, so as to obtain the target extraction image data containing the target detection box; S4, fusing the target extraction image data and the semantic description auxiliary data to obtain a fusion frame of each target; S5, according to the fusion frame, using a perspective projection algorithm, calculate the coordinate position information of each target in the pre-built BIM three-dimensional space, obtain the three-dimensional world coordinates of the target, and map the two-dimensional coordinate information of each target to the three-dimensional space; S6, uses the Dynamo visual programming language to input the 3D world coordinates of each target into the pre-built BIM model to achieve the construction and real-time update of the multimodal digital twin scene.

2. According to claim 1, a multimodal digital twin scene construction method based on image semantic fusion is characterized in that ,The image data set is obtained by taking images of the construction site by ,camera and sensor; the text data set is constructed according to ,construction-related regulations, construction records, safety instructions and the text ,detailed description information corresponding to the image data set.

3. According to claim 1, a multimodal digital twin scene construction method based on image semantic fusion is characterized in that , the preprocessing includes: Performing cropping, resizing and normalization resetting operations on the image data of the image data set; Based on the GPT model, dangerous words are extracted and reorganized on the text data of the text data set.

4. According to claim 1, a multimodal digital twin scene construction method based on image semantic fusion is characterized in that ,The target detection algorithm is an improved target detection model that introduces the Transformer attention mechanism, as follows: Adopt an improved feature extractor, that is, introduce the Transformer module into the feature extractor of the object detection model, and use position encoding to maintain spatial information; Adopt an improved cross-layer attention module, that is, design a cross-layer attention module to allow interaction and information transfer between features of different layers; An improved target detection head is adopted. That is, in the target detection head, the attention mechanism of Transformer is introduced to enhance the focus on the detection target; and the attention weight is used to dynamically weight different feature maps to improve the detection accuracy.

5. According to claim 1, a multimodal digital twin scene construction method based on image semantic fusion is characterized in that ,The fusion of the target extraction image data and the semantic description auxiliary data is specifically: according to the target extraction image data and the semantic description auxiliary data, the target frame of the target extraction image data is fused with the text frame of the semantic description auxiliary data to obtain a more accurate fusion frame of the target, and the steps are as follows: Step 1: Define any target box , , , , and the corresponding text box , , , ; in, , Target box The coordinates of the upper left corner, , Target box The coordinates of the lower right corner; , For text box The coordinates of the upper left corner, , For text box The coordinates of the lower right corner; Step 2: Calculate the target box With text box The coordinates of the overlapping parts , , , , the coordinate values ​​are: =max( , ); ; min( , ); ; like or , then the target box With text box No overlap; Otherwise, if they overlap, calculate the overlapping area for: （ ）； Step 3: Define a fusion box F , , , , the fusion box F is the weighted average position of the area of ​​the target box A and the area of ​​the text box B: =frac( + )( + ); ( + )； )( + )； ( + )； in, (.) represents a fraction, which is the Fraction class of the Python language, used to accurately represent and process fractions; Represents the area of ​​the target box A; Represents the area of ​​text box B.

6. A multimodal digital twin scene construction method based on image semantic fusion according to claim 5, characterized in that ,Based on Python language, according to the calculated fusion frame of each target, the perspective projection algorithm is used to map it to the three-dimensional space, and the 3D space coordinate position information of each target is calculated, specifically: The angle between the camera and the two edges of the object passing through the maximum range of the lens is called the field of view FOV. If the object exceeds the range of the field of view FOV, it will not be captured by the camera. The angle of the field of view is , i.e. the diagonal field of view; According to the diameter of the imaged object on the diagonal of the rectangular photosensitive surface, the diagonal field of view is used. Calculate the object distance s. Since the field of view of the camera is fixed and known, and the range that can be captured is also related to the focal length of the lens, the formula for the object distance s that the camera can capture is: ； in, is the diameter of the imaged object within the visible range of the lens, c is the point of the target image, and is also the intersection of the optical axis and the reference plane; The two-dimensional image captured by the camera is a projection of the three-dimensional space on a plane. Based on the camera posture and the coordinates of the target object in the image, that is, the fusion frame of the target, the position of the target object in the real world is obtained; Assume that the three-dimensional coordinates of the camera are point H (0, h, 0), and the pitch angle between H and the horizontal direction is θ; the reference plane is the plane where the target object is located, and is set on the same plane as the center of the target object. The object distance s is the object distance that the camera can capture, that is, the optical axis, and point c ( , 0, 0) is the point of the target image, which is also the intersection of the optical axis and the reference plane. The coordinates of point c and the trigonometric function relationship of angle θ are obtained as follows: ； ； According to the perspective projection principle, the image captured by the camera is perpendicular to the optical axis, and the projection on the reference plane is a trapezoid, so the midpoint of the target image is set to point c; then, for any two-dimensional coordinate point Q in the target image ( , ), whose value is obtained by measuring transformation, and the corresponding three-dimensional coordinates of Q in the three-dimensional coordinate system are calculated as follows: , the formula is: ； So as to obtain the passing point The equation of the straight line l between and H is: ； Then the coordinates of the intersection point P of the straight line l and the reference plane are ( , , ), the three-dimensional coordinate formula of point P is: ； Thus, the two-dimensional coordinate point Q is mapped to the corresponding world coordinate point P in the digital twin scene; The two-dimensional coordinates of each target in the image are converted into three-dimensional coordinates; then the three-dimensional coordinates are transformed by the perspective projection of the camera to obtain the world coordinates of all targets mapped to the digital twin scene.

7. The method for constructing a multimodal digital twin scene based on image semantic fusion according to claim 6 is characterized in that , further comprising: optimizing the error of the perspective projection algorithm by using an average error distance loss function, according to the projected target object world coordinate position P captured by the set camera and the three-dimensional coordinates of the target object The measurement accuracy between them is calculated, and the Euclidean distance is used to calculate the mean error distance loss MED. By calculating the Euclidean distance of each target object, the accuracy of the perspective projection algorithm is obtained. The specific calculation formula is: ； ； in: represents the Euclidean distance of the i-th target; N represents the total number of samples; MED value and The smaller the value, the higher the accuracy of the perspective projection algorithm.

8. The method for constructing a multimodal digital twin scene based on image semantic fusion according to claim 1 is characterized in that ,For the natural language processing model and the target detection algorithm, a contrast loss function is used for optimization learning to fine-tune the model parameters.

9. A multimodal digital twin scene construction device based on image semantic fusion, characterized in that: include: Data acquisition and preprocessing module, used to obtain image data sets and text data sets of construction sites and perform preprocessing; The semantic description module is used to input the preprocessed image dataset and text dataset into the natural language processing model for text semantic description, and output the logical and structured text sentences corresponding to each image, so as to obtain semantic description auxiliary data; wherein, the natural language processing model is an improved natural language processing model that introduces the Transformer attention mechanism, which is as follows: An improved image encoder is used, that is, the image encoder of the natural language processing model is replaced with an image encoder based on Transformer; the input image data is divided into multiple image blocks of fixed size, and after passing through the linear embedding layer, it is sent to the Transformer encoder to extract the target objects existing in the corresponding scene in the image data; An improved text encoder is used, that is, the text encoder of the natural language processing model is replaced with a text encoder based on Transformer. The input text data is converted into word vectors by the embedding layer and then sent to the Transformer encoder to match the content of the image data with the text data and output the logical and structured text sentences corresponding to each image. The target extraction module is used to input the preprocessed image data set into the target detection algorithm for target classification detection, and output each target object image selected by the classification box, so as to obtain the target extraction image data containing the target detection box; An image semantic fusion module, used to fuse the target extraction image data with the semantic description auxiliary data to obtain a fusion frame of each target; A perspective mapping module is used to calculate the coordinate position information of each target in the pre-built BIM three-dimensional space according to the fusion frame by using a perspective projection algorithm, and obtain the three-dimensional world coordinates of the target, so as to map the two-dimensional coordinate information of each target into the three-dimensional space; The dynamic digital twin scene module is used to input the three-dimensional world coordinates of each target into the pre-built BIM model using the Dynamo visual programming language, thereby realizing the construction and real-time update of multimodal digital twin scenes.

Images