Intelligent auxiliary decision-making method and system for bridge construction progress based on multi-modal visual language model

Abstract

The invention discloses an intelligent auxiliary decision-making method and system for a bridge construction progress based on a multi-modal visual language model, belongs to the technical field of bridge engineering management, and solves the problems that when an existing multi-modal model is used for integrating static scene data and dynamic real-time data of a bridge construction site, the fusion degree is low, and the integration efficiency is high. And therefore, the problems of information incompleteness and cognition deficiency can be solved. The method comprises the following steps: acquiring bridge construction progress data, preprocessing the data, and training a constructed multi-modal bridge construction decision analysis model based on image-text information fusion; preprocessing bridge construction progress data acquired in real time, inputting the preprocessed data into the trained image-text information fused multi-modal bridge construction decision analysis model for analysis, and constructing a unified fusion expression model for feeding back the construction progress and quality in real time according to an instruction normalized visual model required by multi-level user tasks; and obtaining a bridge construction progress scene three-dimensional model. The method is used for intelligent auxiliary decision making of the bridge construction progress.

Description

Technical Field

[0001] A method and system for intelligent auxiliary decision-making of bridge construction progress based on a multimodal visual language model, which are used for intelligent auxiliary decision-making of bridge construction progress, belong to the technical field of bridge engineering management. Background Art

[0002] Bridge construction progress monitoring and management is an important part of the engineering construction process. With the development of information technology, project management and construction monitoring in bridge construction are increasingly relying on intelligence and data. Technologies such as remote real-time monitoring and data analysis can effectively improve the information level of bridge structure monitoring and construction process management to ensure the quality and safety of the project. However, in the process of bridge construction, due to its practical task characteristics such as long construction period, multi-field technical collaboration, and heavy engineering responsibility, real-time monitoring of construction progress and quality has become an extremely important demand. Although existing artificial intelligence technologies have been applied in construction monitoring, they face many challenges. Most of the current methods rely on independent image recognition or simple data analysis. These methods have problems such as insufficient real-time data processing capabilities, difficulty in comprehensive analysis of image and text data, insufficient understanding of professional knowledge and scene context, and challenges in data security and privacy protection, and cannot provide effective decision support and dynamic early warning capabilities. These traditional AI systems often cannot achieve dynamic monitoring and early warning of the entire construction process, which limits the actual application effect in complex engineering environments. Therefore, how to efficiently and accurately monitor the progress of bridge construction and assist decision-making is an important issue.

[0003] The multimodal visual language model is currently the research focus in the field of multimodal artificial intelligence. Using AI models to conduct real-time monitoring and intelligent analysis of the entire bridge construction process can help project managers intelligently handle the current construction progress, which is very important for the overall bridge construction project. However, the existing technology has the following technical problems:

[0004] 1. The existing multimodal model has a low degree of integration when integrating static scene data and dynamic real-time data of bridge construction sites, resulting in incomplete information and lack of cognition, which limits the model's ability to comprehensively analyze construction progress and quality;

[0005] 2. During the bridge construction process, real-time data is crucial for monitoring and prediction. However, the existing AI models rely on on-site staff to manually record and upload data information in terms of data update frequency and efficiency. They are unable to capture the latest status and changes of the construction site in a timely manner, thus affecting the ability to respond to potential problems in a timely manner and support decision-making.

[0006] 3. Traditional multimodal AI models are deficient in their information cognition. Even though they can integrate multiple data sources, they perform poorly in converting data into understandable and explainable information. The cognitive scope of broad generative AI models is limited to traditional public knowledge information. They lack in-depth professional experience and knowledge in the field of bridge construction. They lack sufficient accuracy and depth in their decision-making capabilities for bridge construction progress and quality analysis, and are unable to provide operational recommendations and safety tips. Summary of the invention

[0007] The purpose of the present invention is to provide an intelligent auxiliary decision-making method and system for bridge construction progress based on a multimodal visual language model, so as to solve the problem that the existing multimodal model has a low degree of fusion when integrating static scene data and dynamic real-time data of the bridge construction site, resulting in incomplete information and lack of cognition, which limits the model's ability to comprehensively analyze the construction progress and quality.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] An intelligent auxiliary decision-making method for bridge construction progress based on a multimodal visual language model includes the following steps:

[0010] Step 1: Obtain and pre-process the bridge construction progress data, including engineering data, construction data, monitoring images and on-site data;

[0011] Step 2: construct a multimodal bridge construction decision analysis model based on image-text information fusion, and train it with the bridge construction progress data obtained through preprocessing;

[0012] Step 3: Preprocess the real-time acquired bridge construction progress data and input it into the trained image-text information fusion multimodal bridge construction decision analysis model to integrate the static scene data and data dynamic analysis of the bridge construction site, and construct a unified fusion expression model for real-time feedback of construction progress and quality based on the instructions of multi-level user task requirements, that is, to obtain a three-dimensional model of the bridge construction progress scene.

[0013] Further, the specific steps of step 2 are:

[0014] Step 2.1, construct a bridge construction progress target monitoring model based on the improved YOLO-Bridge3D and a three-dimensional information decoding predictor and train it through the pre-processed bridge construction progress data to obtain bridge construction structure detection;

[0015] Step 2.2, construct a multimodal private domain VectorStore structure database and a bridge construction expert LLM model based on dynamic monitoring data and private knowledge information, and train the detected bridge construction structure to obtain knowledge-guided construction status analysis, and finally obtain a trained image-text information fusion multimodal bridge construction decision analysis model.

[0016] Furthermore, the improved YOLO-Bridge3D bridge construction progress target monitoring model in step 2.1 is based on the Yolov5s model, adding a CBAM channel attention mechanism module to its backbone network, and adding a Dice Loss function to the head network to obtain boundary information, including the structural type of the bridge, spatial morphological constraints and boundary discrete information.

[0017] Furthermore, the improved YOLO-Bridge3D bridge construction progress target monitoring model adds CBAM channel attention mechanism modules between the first Conv2D layer and the second Conv2D layer, between the second Conv2D layer and the third Conv2D layer, between the third Conv2D layer and the fourth Conv2D layer, and after the SPPF layer in the backbone network of the Yolov5s model;

[0018] The formula of the Dice Loss function is:

[0019]

[0020] Among them, P i and G i They represent the predicted result and the true label value at the i-th pixel respectively, and N represents the total number of pixels.

[0021] Further, the CBAM channel attention mechanism module includes a channel attention mechanism module that receives the input features of the previous layer output, and a spatial attention mechanism module that receives the result of channel multiplication of the output of the channel attention mechanism module and the input features, and the result output after the dimension multiplication of the input and output of the spatial attention mechanism module is the output of the CBAM channel attention mechanism module;

[0022] The channel attention mechanism module includes a global maximum pooling layer and a global average pooling layer that receive the input features of the previous layer output in parallel, a multi-layer perceptron connected to the global maximum pooling layer and the global average pooling layer, and a Sigmoid activation function layer that generates a channel attention map based on the output summation result of the multi-layer perceptron. The channel attention map is the output of the channel attention mechanism module;

[0023] The spatial attention mechanism module includes a global maximum pooling layer, a global average pooling layer, a 7*7 convolution layer and a Sigmoid activation function layer, which sequentially receive the output of the channel attention mechanism module and the multiplication of the input feature. The output of the Sigmoid activation function layer is a spatial attention map, that is, the output of the spatial attention mechanism module, wherein the inputs of the global maximum pooling layer and the global average pooling layer are channel-concatenated and used as the input of the 7*7 convolution layer.

[0024] Furthermore, it is characterized in that the specific implementation steps of the three-dimensional information decoding predictor are:

[0025] Step 2.11: First, use the Visvalingam-Whyatt algorithm to simplify the feature information of the boundary discrete information output by the improved YOLO-Bridge3D bridge construction progress target monitoring model, and match the simplified feature information. The specific steps are as follows:

[0026] First, calculate the area of ​​each triangle formed by three consecutive points. The calculation formula is:

[0027]

[0028] Among them, (x1, y1), (x2, y2), (x3, y3) are the coordinates of the three vertices of the triangle, and the point sequence P1, P2, P3...P m , for each point P l , l takes the value from 2 to m-1, then the calculation is done by P l-1 ,P l ,P l+1 The area of ​​the triangle formed;

[0029] Then, find the triangle with the smallest area from all the calculated triangles and remove the middle point P of the smallest triangle. l , and remove P l After the point is found, update the areas of the two triangles adjacent to the smallest triangle;

[0030] Next, the previous step is repeated, and the middle point of the triangle with the smallest area is removed in each iteration until the preset simplification threshold is reached, that is, the simplified feature information is obtained;

[0031] Finally, based on the simplified feature information, the FLANN matcher is used to quickly find the nearest neighbor actual boundary discrete points in the high-dimensional space, and the actual boundary discrete points are matched and calibrated with the expected reference points of the theoretical model;

[0032] Step 2.12, using GPS geographic information to match the bridge construction progress data of the bridge construction site, and clarifying the design form of the bridge construction site based on the construction technology used at the bridge construction site and the structural type of the bridge, and then using the spatial form constraints of the bridge to establish the bridge construction site constraint conditions;

[0033] Step 2.13, based on the constraints and matching results, the photographic scale conversion coefficient is calculated using the perspective projection principle from three-dimensional space to two-dimensional plane, and the size information of the changing dimension is deduced from the projection of a fixed two-dimensional plane in the real three-dimensional space on the image plane, that is, the three-dimensional structural size information of the bridge construction progress is decoded.

[0034] Furthermore, the specific steps of constructing the multimodal private domain VectorStore structure database in step 2.2 are:

[0035] On the basis of dynamic monitoring data and private knowledge information, a non-structured relational database is established by using word vectorization and format conversion methods. That is, after obtaining the number of bridge construction progress vectors that need to be stored and the dimension of each vector, a vector database with nearest neighbor search as the core is established. Then, the inverted file indexing technology is used to find the data point closest to the output result of the three-dimensional information decoding predictor in the relational database.

[0036] The data points obtained from the query are combined with the information input by the user to form a prompt input template and input into the trained bridge construction expert LLM model for knowledge-guided construction status analysis, including analyzing the stage of bridge construction, judging the time difference between the current construction progress and the theoretical construction progress, giving whether the current construction status is normal, ahead of schedule or delayed, and analyzing and judging how many days the current construction period differs from the actual construction period.

[0037] Furthermore, the specific steps of fine-tuning the improved language model ChatGLM based on P-tuning in step 2.2 to obtain the bridge construction expert LLM model based on dynamic monitoring data and private knowledge information are as follows:

[0038] First, the pre-trained language model ChatGLM is initialized, that is, the language model ChatGLM is initialized to have a specific hidden layer size H and vocabulary size |V|, wherein the hidden layer size H refers to the number of neurons in the hidden layer of the language model ChatGLM or the dimension of the feature vector, and the weight matrix and bias vector of the language model ChatGLM are further initialized according to the hidden layer size H set by the initialization, and the vocabulary size |V| refers to the number of words or subword units in the vocabulary of the language model ChatGLM that contain information in the field of bridge construction;

[0039] Next, we construct and use the word information provided by the vocabulary |V| to form a template T using discrete prompt tags. The template T organizes the input discrete data into a series of text tags, denoted by T = {[D 0:i ],x,[D (i+1):j ],y,[D (j+1):k ]}, where [D i ] represents a discrete prompt mark, i and j both represent the location of the discrete prompt mark, [D 0:i ] represents a discrete sequence of prompt tokens from index 0 to i, [D (i+1):j ] represents a discrete prompt tag sequence from index i+1 to j, [D (j+1):k ] represents the discrete prompt tag sequence from index j+1 to k, that is, it includes all discrete prompt tags from the h+1th position to the kth position, where the discrete data includes the annotation data x and the label y;

[0040] Secondly, the discrete data is applied to the template T;

[0041] By keeping the language model ChatGLM parameters frozen, we use an additional embedding function f(P i ):{h0,...,h j ,e(x),h i+1 ,...,h j ,e(y),h j+1 ,...,h K} Make a mapping template to map the discrete prompt tags and discrete data together to the pre-trained embedding layer to construct a new prompt template T = {[P 0:i ],x,[P (i+1):j ],y,[P (j+1):k ]}, the fine-tuned language model ChatGLM is obtained, that is, the bridge construction expert LLM model with dynamic monitoring data and private knowledge information is obtained, which is used for training with the Prompt input template to obtain {e(D0)...e(D j ),e(x0),...,e(x n ),e(D k )}, where e represents the embedding matrix, P i is the i-th continuous prompt embedding, h0 represents the hidden layer state at the 0th layer position, h j represents the hidden layer state of the jth layer, h i+1 represents the hidden layer state at the i+1th layer position, h K represents the hidden layer state at the Kth layer position, e(x) and e(y) represent the embedded representations of different labeled data x and labels y respectively, [P 0:i ] represents the discrete hint tag embedding of the 0th to ith parts, [P (i+1):j] represents the discrete hint tag embedding of the i+1th to jth parts, [P (j+1):k ] represent the discrete cue tag embeddings from the j+1th to the kth part, e(D0), e(D j )、e(x0)、e(x n )、e(D k ) represent the embedded representations of different discrete prompt tag sequences, respectively. The parameter freezing of the language model ChatGLM includes the freezing of the main weights, that is, the freezing of the weights and bias parameters of the self-attention layer.

[0042] A bridge construction progress intelligent auxiliary decision-making system based on a multimodal visual language model comprises a processor and a memory storing a computer program, wherein the processor executes the computer program to implement the bridge construction progress intelligent auxiliary decision-making method based on a multimodal visual language model as described in the claim.

[0043] Compared with the prior art, the advantages of the present invention are:

[0044] 1. The CBAM channel attention mechanism module added to the improved YOLO-Bridge3D bridge construction progress target monitoring model in the present invention can suppress invalid information areas and enhance the ability to extract feature-rich areas. By adding the feature area mask information obtained by the DiceLoss loss function, the spatial morphological constraints of the bridge structure are used to establish a 3D detection technology suitable for construction variable sizes, solve the problem that the original Yolo series algorithm cannot extract construction progress and quality from monitoring cameras (that is, when integrating static scene data and dynamic real-time data at the bridge construction site, the degree of fusion is low, resulting in incomplete information and cognitive loss), and achieve low-cost, high-efficiency and high-precision bridge construction dynamic data acquisition, which solves the problem that the original Yolo series model can only realize target detection but cannot determine the 3D size information (length, width and height) of the target, that is, any existing Yolo model cannot determine the 3D size information (length, width and height) of the target;

[0045] Second, the present invention constructs a bridge construction expert LLM model based on dynamic monitoring data and private knowledge information, and establishes an expert model in the vertical field of bridge construction based on the methodology of P-Tuning fine-tuning ChatGLM, so as to realize a bridge construction expert LLM model that can analyze and answer bridge construction engineering problems. At the same time, the image monitoring analysis results and bridge engineering project professional knowledge information are used to quantify, transform and establish private domain data, which not only ensures the reliability, security and real-time nature of enterprise data, but also effectively solves the model hallucination problem and the problem of insufficient general model capabilities, that is, it can timely capture the latest status and changes of the construction site, and provide an efficient and professional bridge construction expert LLM model for actual bridge engineering;

[0046] 3. The present invention is based on the modular normalization fusion technology of multimodal data, combined with the improved YOLO-Bridge3D bridge construction progress target monitoring model and the bridge construction expert LLM model, to realize intelligent monitoring and decision-making of the bridge construction process, and reflect the output results of all artificial intelligence models on the bridge construction visualization model, so as to show the artificial intelligence analysis results to users from a more intuitive and three-dimensional perspective, and provide construction visualization analysis reports and intelligent decision-making auxiliary questions and answers with professional knowledge and experience, which can vividly depict the current bridge construction status and give feasible decision-making suggestions. All data and systems can be realized in the local area network, and meet the data security and practical application needs of enterprises in an efficient and intelligent way, that is, the performance is good when converting data into understandable and explainable information, and the cognitive scope is not limited to traditional public knowledge information, and can go deep into the professional experience knowledge in the field of bridge construction, and have sufficient accuracy and depth in the decision-making ability of bridge construction progress and quality analysis, and can provide operational suggestions and safety tips;

[0047] 4. The image-text aligned multimodal model in the prior art only improves the detection means or replaces the detection object, and requires image and text alignment, the purpose is to make the detection result more accurate and reliable, and the detection target more diversified. However, the present invention is different (no image and text alignment is required). The present invention starts from the perspective of engineering application, and what is to be detected is the quality and progress of the bridge. The ordinary multimodal detection method (image-text aligned multimodal model) cannot realize the content of three-dimensional detection, but the present invention can achieve it and provide richer data information, that is, the transformation from 2D anchor box to 3D bonding box. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 It is the overall research idea diagram of the present invention, wherein Yolo3D represents the improved YOLO-Bridge3D bridge construction progress target monitoring model, ChatGLM represents the bridge construction expert LLM model, Splitter represents the document segmentation technology, and Embadding represents the vectorization technology;

[0050] Figure 2It is a structural diagram of the improved YOLO-Bridge3D bridge construction progress target monitoring model in the present invention, wherein Focus represents the input layer, Conv2D represents the convolution layer, CBAM represents the CBAM channel attention mechanism module, SPPF represents the pooling pyramid layer, Concat represents the fully connected layer, Upsample represents the upsampling layer, C3 represents the residual layer, and Dice Loss represents the Dice Loss function;

[0051] Figure 3 It is the structural diagram of the CBAM attention mechanism module, where a represents the structural diagram of the channel attention mechanism module; b represents the structural diagram of the spatial attention mechanism module; c represents the structural diagram of the CBAM attention mechanism module;

[0052] Figure 4 3D information decoding predictor structure flow chart of the present invention, wherein h1 and h2 represent the height in T1-construction size and T2-construction size respectively, K represents the photography ratio conversion coefficient, T1 represents the previous construction time, T2 represents the subsequent construction time, category represents the structural category, and category2 represents the bridge pier;

[0053] Figure 5 It is a schematic diagram of the principle of fine-tuning the language model ChatGLM based on P-Tuning in the present invention;

[0054] Figure 6 It is a flow chart of the VectorStore structure database stored in the private domain of the present invention, wherein VectorDatabase represents the name of the vector database;

[0055] Figure 7 It is a flow chart of the overall structure of the system of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] The overall research idea is as follows Figure 1As shown in the figure, it mainly includes three aspects: the improved YOLO-Bridge3D bridge construction progress target monitoring algorithm, the private domain knowledge information storage technology based on the quantitative fusion of multimodal data, and the human-computer interaction and three-dimensional expression system based on the fine-tuning LLM model. In the first aspect, based on the traditional target detection model of the original Yolo series, the CBAM channel attention mechanism module is added to its backbone network to suppress invalid information areas and enhance the ability to extract feature-rich areas; secondly, the new Dice Loss loss function is introduced to establish a new target detection head branch, and finally the convolutional neural network (multimodal bridge construction decision analysis model with image-text information fusion) is used to extract the boundary information of the bridge structure, and the information decoding predictor of the three-dimensional structure size is established in combination with the spatial geometric morphology constraints to output the 3D size information of the bridge structure. Secondly, we build an LLM model of bridge construction experts based on dynamic monitoring data and private knowledge information. First, we use the image monitoring analysis results and professional knowledge information of bridge engineering projects to quantify and transform them, and then establish a multimodal private domain VectorStore structure database. Secondly, we use the P-tuning algorithm to fine-tune the model to reduce the number of parameters. Finally, we design a storage and retrieval data structure tree for local text word vectors to build an intelligent question-answering model for real-time analysis of bridge construction quality and progress. Thirdly, based on the above model, we establish a multimodal bridge construction decision analysis model based on image-text information fusion. By integrating the static scene data and dynamic data analysis of the bridge construction site, we design a normalized visualization model for instructions according to the task requirements of multi-level users, and build a unified fusion expression model for real-time feedback on construction progress and quality. The specific steps are:

[0058] Obtain and preprocess the bridge construction progress data, including engineering data, construction data, monitoring images and on-site data. Preprocessing includes parsing the bridge knowledge base to generate document data, establishing standardized documents and segmenting them into text blocks; using monitoring cameras and GPS geographic information to extract key frames, regularly establishing photo images with actual spatial position coordinates, and establishing bridge construction image data through data cleaning, error correction, distortion correction and image enhancement.

[0059] A multimodal bridge construction decision analysis model based on image-text information fusion is constructed and trained with the bridge construction progress data obtained through preprocessing; the specific steps are as follows:

[0060] The Yolo model based on multi-layer convolutional neural network for bridge structure recognition has achieved good results in the field of target detection. However, in actual engineering applications, due to the large differences in structural feature information at different stages of bridge construction, the large number of construction structure types, and the huge differences in process and process indicators, it is difficult to accurately locate the three-dimensional size of the target using monitoring images, which hinders the promotion and application of the algorithm model in actual engineering. The existing YoloV5S model extracts the feature information of the construction structure through the convolution layer of the backbone network, aggregates the features through the neck network, and finally predicts the head to locate, classify and score the information to complete the target structure detection. However, in actual use, the single convolution layer has weak ability to extract structural feature information for high-pixel and large-scale targets, and cannot effectively distinguish the detailed structural feature information of the target. In addition, the target classification and positioning methods used by the prediction head cannot meet the needs of actual three-dimensional size positioning, and do not have the ability to obtain three-dimensional construction feature information of bridge structures. Therefore, in response to the above problems, based on the bridge BIM construction information and GPS geographic information data, an improved image feature information extraction network (i.e., the improved YOLO-Bridge3D bridge construction progress target monitoring model) and a three-dimensional size information decoding predictor (i.e., the three-dimensional information decoding predictor) are established to solve the problem of lack of three-dimensional information of bridge structure in the actual construction monitoring process. That is, the bridge construction structure detection is obtained by training based on the improved YOLO-Bridge3D bridge construction progress target monitoring model and the three-dimensional information decoding predictor and the bridge construction progress data obtained by preprocessing. Specifically: Based on the bridge BIM construction information and GPS positioning information, this case designs an image feature information extraction network for the characteristics of bridge construction structure, and proposes a CBAM feature attention mechanism module using spatial attention and channel attention to enhance the feature expression ability of the backbone network. On the basis of the branch of the detection head for target positioning and classification, the Dice Loss function is added to establish the target image segmentation branch, and the input image data is converted into the boundary information of the bridge structure characteristics, including discrete boundary information and structure type, and the spatial morphological constraints of the construction structure characteristics are obtained for the three-dimensional information decoding predictor to decode the actual three-dimensional structure size information. The improved YOLO-Bridge3D bridge construction progress target monitoring model is shown in Figure 1. Figure 2 shown.

[0061] This case uses a lightweight CBAM channel attention mechanism module to perform Attention calculations in the channel and spatial dimensions. Through a given intermediate feature map, attention is inferred along two independent dimensions, spatial and channel. Finally, the attention map is adaptively matched with the input feature map to achieve end-to-end extraction and training of feature information at different scales. The CBAM channel attention mechanism module consists of a channel attention mechanism module and a spatial attention mechanism module. Its structure is as follows: Figure 3 shown.

[0062] The channel attention mechanism module uses the global maximum pooling layer MaxPool and the average pooling layer AvgPool to operate the input feature map to obtain a low-dimensional feature map, and then uses the output of the multi-layer perceptron MLP to perform an Element-wise addition operation. The calculation formula is as follows. Finally, the required channel attention feature map is generated based on the Sigmoid activation function layer.

[0063] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) (1)

[0064] Among them, σ represents the Sigmoid activation function.

[0065] The spatial attention mechanism module uses the result of multiplying the input features of the channel attention mechanism module with the channel attention feature map as the input feature map, and performs global maximum pooling layer and average pooling layer operations in sequence. The two feature maps output by the global maximum pooling layer and the average pooling layer are respectively subjected to channel splicing operations, and then a 7×7 large convolution kernel is used for dimensionality reduction. The dimensionality reduction formula is as follows. Finally, a spatial attention mechanism map is generated based on the sigmoid activation function layer.

[0066] M c (F) = σ(f 7×7 ([AvgPool(F);MaxPool(F)])) (2)

[0067] Among them, f 7×7 It represents a convolution operation with a convolution kernel of size 7×7.

[0068] In the CBAM channel attention mechanism module, the "multiplication" symbol represents the element-by-element multiplication operation. This operation enables the model to assign higher weights to specific channels or spatial positions, thereby highlighting important features and suppressing irrelevant information, which can effectively improve the model's ability to capture important information and improve the performance of the overall network.

[0069] Based on the original Yolov5S model detection head, a new prediction branch structure is constructed by introducing the Dice Loss function, judging the set similarity between the predicted area branch and the real area, and constructing a branch structure with the gradient propagation of the improved YOLO-Bridge3D bridge construction progress target monitoring model. The formula of its loss function is as follows:

[0070]

[0071] Among them, P i and G i They represent the predicted result and the true label value at the i-th pixel respectively, and N represents the total number of pixels.

[0072] Based on the above improvements, this case realizes adaptive weight training for feature channel information and spatial position information of different scales. While enhancing the model's feature expression ability and decision accuracy, a new pixel loss function is added as an additional detection branch to extract more detailed discrete boundary information. The original detection head branch also provides structural category information (i.e., structural type), discrete boundary information and spatial morphology constraints for the subsequent three-dimensional information decoding predictor.

[0073] In this case, based on the discrete boundary information, structural category information and spatial morphological constraints provided by the improved YOLO-Bridge3D bridge construction progress target monitoring model, a three-dimensional information decoding predictor is established. The feature information is simplified by the Visvalingam-Whyatt algorithm, and redundant noise feature information is removed. Then, the structural category information is used to clarify the design form of the bridge structure, and the spatial geometric form of the bridge structure is used to establish constraints. The photographic scale conversion coefficient is derived and perspective projection changes are performed to decode the 3D size information of the actual bridge structure. The structural process of its design is as follows: Figure 3 The specific implementation steps of the three-dimensional information decoding predictor are as follows:

[0074] First, the Visvalingam-Whyatt algorithm is used to simplify the feature information of the boundary discrete information output by the improved YOLO-Bridge3D bridge construction progress target monitoring model, and then the simplified feature information is matched. The specific steps are as follows:

[0075] First, calculate the area of ​​each triangle formed by three consecutive points. The calculation formula is:

[0076]

[0077] Among them, (x1, y1), (x2, y2), (x3, y3) are the coordinates of the three vertices of the triangle, and the point sequence P1, P2, P3...P m , for each point Pl , l takes the value from 2 to m-1, then the calculation is done by P l-1 ,P l ,P l+1 The area of ​​the triangle formed;

[0078] The Visvalingam-Whyatt algorithm is an algorithm for simplifying discrete boundary feature information. It reduces data complexity by minimizing the number of points on the contour while maintaining the overall characteristics of the discrete boundary as much as possible.

[0079] Then, find the triangle with the smallest area from all the calculated triangles and remove the middle point P of the smallest triangle. l , and remove P l After the point is found, update the areas of the two triangles adjacent to the smallest triangle;

[0080] Next, the previous step is repeated, and the middle point of the triangle with the smallest area is removed in each iteration until the preset simplification threshold is reached, that is, the simplified feature information is obtained;

[0081] Finally, based on the simplified feature information, the FLANN matcher is used to quickly find the nearest neighbor actual boundary discrete points in the high-dimensional space, and the actual boundary discrete points are matched and calibrated with the expected reference points of the theoretical model;

[0082] Use GPS geographic information to match the bridge construction progress data at the bridge construction site, and clarify the design form of the bridge construction site based on the construction technology used at the bridge construction site and the structural type of the bridge, and then use the spatial form constraints of the bridge to establish the constraints of the bridge construction site;

[0083] Based on the constraints and matching results, the photographic scale conversion coefficient is calculated by using the perspective projection principle from three-dimensional space to two-dimensional plane, and the size information of the changing dimension is derived from the projection of a fixed two-dimensional plane in the real three-dimensional space on the image plane, so as to decode the three-dimensional structural size information of the bridge construction progress. That is, according to the technical characteristics of the bridge construction process, the linear scale coefficient is calculated based on the constraints and matching results using the perspective projection principle from the real three-dimensional space to the two-dimensional plane of the actual engineering structure of the bridge construction progress, and at the same time, the size information of the changing dimension can be derived based on the projection of a fixed two-dimensional plane in the real three-dimensional space on the image plane, thereby decoding the three-dimensional structural size information of the bridge structure during the dynamic change of construction.

[0084] The existing large language artificial intelligence model has the ability to answer basic knowledge questions of engineers in general fields, but when targeting specific bridge construction scenarios, the pre-trained model is not sensitive enough to the specific terms of the bridge construction industry, resulting in model hallucinations, wrong answers and unclear cognition in the actual reasoning and analysis process. At the same time, the generalization ability of specific engineering tasks is very weak, and it is unable to access real-time dynamic resource data, resulting in an inability to adapt to the actual needs of intelligent analysis of bridge construction. In response to the above problems, this case designed a language model ChatGLM based on P-Tuning fine-tuning and improvement. By obtaining a specific bridge construction expert knowledge data set for localized training, the model can be adapted to actual task needs more quickly, thereby reducing the consumption of computing resources and operating costs in actual use. At the same time, the multimodal real-time data information is converted into unstructured information of a unified mode by using word vectorization and format conversion technology, and then further constructed and stored in the multimodal private domain VectorStore structure database, and the user's questions and dialogue history are used together to establish the PromptTemplate structured template, which plays a role in efficiently constructing prompts for bridge construction analysis tasks. That is, a multimodal private domain VectorStore structure database and a bridge construction expert LLM model based on dynamic monitoring data and private knowledge information are constructed, and the bridge construction structure obtained by detection is trained to obtain knowledge-guided construction status analysis, including analyzing the stage of bridge construction, judging the time difference between the current construction progress and the theoretical construction progress, giving whether the current construction status is normal, advanced or delayed, and analyzing and judging the difference between the current construction period and the actual construction period. Finally, a trained multimodal bridge construction decision analysis model with image-text information fusion is obtained, which is specifically:

[0085] Based on the private domain dataset of bridge construction expert experience knowledge, this case designs a bridge construction expert LLM model for the bridge construction process, and proposes to use P-Tuning fine-tuning technology to improve the ChatGLM model, which is used to enhance the bridge construction expert LLM model's ability to understand specific terms and knowledge in professional fields, improve the model's generalization ability and knowledge analysis ability, avoid the problem of knowledge hallucination in the model, and nest the trained model and deploy it locally to solve the security risk of user information leakage during actual use, and effectively protect the production information security of enterprise users.

[0086] The P-tuning technology used in this case is different from the traditional fine-tuning of the complete model. It only adjusts the input part of the model, so that the model is more suitable for a specific bridge construction professional knowledge data set. It only adjusts a small part of the bridge construction expert LLM model instead of all parameter information, and has the characteristics of high parameter efficiency. First, the Prompt is constructed to establish a series of learnable virtual tokens as the prompt of the model. The virtual tokens of knowledge and experience are initialized and embedded in the input sequence of the model. When processing the input, the model regards the virtual token as part of the sequence, and the token interacts through the self-attention mechanism of the bridge construction expert LLM model. Among them, during the back propagation and optimization process of the training process, only the parameters related to the virtual token will be updated, and the main weights of the bridge construction expert LLM model remain unchanged. The overall process is as follows: Figure 5 As shown, it can effectively reduce the number of parameters that need to be optimized during training and reduce resource consumption.

[0087] Initialize the virtual token of knowledge and experience and embed it into the language model ChatGLM. The input sequence part refers to further introducing the method of continuous prompt embedding on the basis of discrete prompt search to obtain the fine-tuned language model ChatGLM, that is, to obtain the bridge construction expert LLM model of dynamic monitoring data and private knowledge information. The specific steps are:

[0088] Existing LLM models are mostly deployed on cloud servers, using public knowledge data sets for interpretation and analysis. They cannot access private enterprise data information and cannot connect real-time sensor data to artificial intelligence models for instant analysis. There are problems such as slow real-time data interaction, high risk of data leakage, and difficulty in storing multimodal data. Therefore, this patent proposes to use word vectorization and multimodal conversion technology to uniformly convert multimodal data such as commercial concrete test TXT data, steel bar spot check HDF5 data, discrete mud preparation XML data, and casing buried sensor JSON data into Vector data streams and store them in local structured databases. The overall process is as follows: Figure 5 As shown, it supports efficient similarity search of large-scale data sets and effectively protects the security of local data information to avoid the risk of data leakage caused by information transmission.

[0089] Based on the local static private knowledge database and dynamic sensor detection data, this case uses word vectorization and format conversion technology to establish an unstructured relational database, then estimates the number of vectors that need to be stored and the dimension of each vector, and establishes a vector database with nearest neighbor search as the core. By adopting the inverted file index (IVF) technology, it effectively supports the rapid retrieval of high-dimensional data, optimizes the data retrieval process, and ensures that the data points closest to the given query can be quickly found even in a huge data set. Finally, the queried data points and the user input information are made into a Prompt instruction template for reasoning analysis of the trained language model ChatGLM (i.e., the trained bridge construction expert LLM model).

[0090] First, the pre-trained language model ChatGLM is initialized, that is, the language model ChatGLM is initialized to have a specific hidden layer size H and vocabulary size |V|, wherein the hidden layer size H refers to the number of neurons in the hidden layer of the language model ChatGLM or the dimension of the feature vector, and the weight matrix and bias vector of the language model ChatGLM are further initialized according to the hidden layer size H set by the initialization, and the vocabulary size |V| refers to the number of words or subword units in the vocabulary of the language model ChatGLM that contain information in the field of bridge construction;

[0091] Next, we construct and use the word information provided by the vocabulary |V| to form a template T using discrete prompt tags. The template T organizes the input discrete data into a series of text tags, denoted by T = {[D 0:i ],x,[D (i+1):j ],y,[D (j+1):k ]}, where [D i ] represents a discrete prompt mark, i and j both represent the location of the discrete prompt mark, [D 0:i ] represents a discrete sequence of prompt tokens from index 0 to i, [D (i+1):j ] represents a discrete prompt tag sequence from index i+1 to j, [D (j+1):k ] represents the discrete prompt tag sequence from index j+1 to k, that is, it includes all discrete prompt tags from the j+1th position to the kth position, where the discrete data includes the annotated data x and the label y;

[0092] Secondly, the discrete data is applied to the template T;

[0093] By keeping the language model ChatGLM parameters frozen, we use an additional embedding function f(P i ):{h0,...,h j ,e(x),h i+1 ,...,h j ,e(y),hj+1 ,...,h K} Make a mapping template to map the discrete prompt tags and discrete data together to the pre-trained embedding layer to construct a new prompt template T = {[P 0:i ],x,[P (i+1):j ],y,[P (j+1):k ]}, the fine-tuned language model ChatGLM is obtained, that is, the bridge construction expert LLM model with dynamic monitoring data and private knowledge information is obtained, which is used for training with the Prompt input template to obtain {e(D0)...e(D j ),e(x0),...,e(x n ),e(D k )}, where e represents the embedding matrix, P i is the i-th continuous prompt embedding, h0 represents the hidden layer state at the 0th layer position, h j represents the hidden layer state of the jth layer, h i+1 represents the hidden layer state at the i+1th layer position, h K represents the hidden layer state at the Kth layer position, e(x) and e(y) represent the embedded representations of different labeled data x and labels y respectively, [P 0:i ] represents the discrete hint tag embedding of the 0th to ith parts, [P (i+1):j ] represents the discrete hint tag embedding of the i+1th to jth parts, [P (j+1):k ] represent the discrete cue tag embeddings from the j+1th to the kth part, e(D0), e(D j )、e(x0)、e(x n )、e(D k ) represent the embedded representations of different discrete prompt tag sequences, respectively. The parameter freezing of the language model ChatGLM includes the freezing of the main weights, that is, the freezing of the weights and bias parameters of the self-attention layer.

[0094] In this case, construction monitoring camera sensors are used as visual input terminals. First, the input image information, bridge BIM construction design information and GPS geographic information are converted into bridge construction dynamic information, including unstructured information composed of construction progress and structure type monitoring information, through the improved YOLO-Bridge3D bridge construction progress target monitoring model and three-dimensional information decoding predictor. Then, a bridge construction expert LLM model based on dynamic monitoring data and private knowledge information is established according to the knowledge database in the field of bridge construction. By comprehensively analyzing the static data and dynamic information in the construction process, an intelligent analysis of the progress and quality of bridge construction is made, and scientific answers are given according to the needs and problems of enterprise users. That is, a visualization model normalized according to the instructions of multi-level user task requirements is constructed to construct a unified fusion expression model for real-time feedback of construction progress and quality, and a three-dimensional model of the bridge construction progress scene is obtained. Based on the multimodal visual language bridge construction decision analysis model mentioned above, this case further designed a three-dimensional modeling function of the bridge construction site guided by the artificial intelligence model. In the Cesium geographic scene visualization modeling platform, by using the bridge structure dynamic model parameters generated by the Yolo-Bridge3D model and the construction progress difference analysis results given by the ChatGLM-Bridge language model, ChatGLM-Bridge uses the API instructions of the system's internal communication to generate the corresponding program code, and automatically executes the code to call the Primitive framework for three-dimensional modeling of the bridge construction structure, and aligns the model to the actual regional location through GIS geographic scene information, and regularly updates it in real time. Finally, a visual analysis report on the bridge construction progress tracking results is generated according to user needs. The overall system structure process is as follows Figure 6 As shown, on the basis of ensuring the security of construction data, it meets the actual user's needs for intelligent decision-making analysis of real-time construction progress and quality, and improves the level of intelligent bridge construction and scientific decision-making management.

[0095] The computer program included in the prototype system developed in this case is mainly divided into six functional modules, namely, knowledge data module, video data module, progress monitoring module, private domain storage module, human-computer interaction module and three-dimensional expression module. Among them, the knowledge data module and the video data module are to obtain and pre-process the bridge construction progress data. The knowledge data module first generates document data by parsing the bridge knowledge base, establishes standardized documents and divides them into text blocks; the video data module uses monitoring cameras and GPS geographic information to extract key frames, regularly establishes photo images with actual spatial position coordinates, and establishes bridge construction image data through data cleaning, error correction, distortion correction and image enhancement. The progress monitoring module uses the pre-processed data obtained by the knowledge data module and the video data module to train the improved YOLO-Bridge3D bridge construction progress target monitoring model. After training, the input image data (i.e., the pre-processed bridge construction progress data obtained in real time) is subjected to specific dynamic detection, and clear bridge structural categories and boundary discrete information and spatial morphological constraints are output to the three-dimensional information decoding predictor, and the bridge construction progress data at the bridge construction site is matched using GPS geographic information, and based on the construction technology and bridge used at the bridge construction site, the bridge construction progress data is matched with the video data. The structural type of the beam clarifies the design form of the bridge construction site, and then uses the spatial form constraints of the bridge to establish the constraints of the bridge construction site, and uses the constraints to derive the dynamic three-dimensional size information of the bridge construction structure; the private domain storage module uses word vectorization and format conversion methods on the basis of dynamic monitoring data and private knowledge information to establish an unstructured relational database, that is, it is responsible for treating the quantized text block information and the converted dynamic three-dimensional size information as unstructured information, and then storing them in the local database through standard word vectorization, and performing consciousness detection on user questions to generate retrieval instructions (that is, generating Prompt input templates). Finally, the index retrieval mechanism will combine the conversation history to provide safe and accurate data encoding vectors; the human-computer interaction module is mainly based on the fine-tuned improved language model ChatGLM-Bridge, and is used to construct Prompt with the assistance of data encoding vector information of relevant professional knowledge and conversation history. The Template is sent to the trained bridge construction expert LLM model for thinking and analysis, and the results of the intelligent question and answer of bridge construction will be output on the human-computer interaction interface; the three-dimensional expression module constructs program code for parametric modeling according to the API instructions transmitted by the ChatGLM-Bridge language model within the system. The modeling-related parameter information comes directly from the model dynamic parameter information of the visual model and the construction progress difference analysis results of the language model. The model is then positioned, registered and updated on the Cesium modeling platform, and finally a visual analysis report is generated as the bridge construction progress tracking result.

[0096] This case uses visual sensors as eyes, Yolo-Bridge3D visual model as the visual center of the brain, bridge construction knowledge as the memory center of the brain, language model ChatGLM-Bridge as the logical center of the brain, Cesium bridge construction visualization model as the mouth, and the actual decision-making needs of bridge construction engineers as the core to form a complete intelligent monitoring and decision support system for bridge construction. The system can monitor the situation of the construction site in real time, predict and identify potential risks and problems by analyzing integrated multimodal data, and thus provide scientific and accurate decision support. Using advanced artificial intelligence technology, this system not only enhances the real-time monitoring capability of construction progress, but also improves the flexibility and efficiency of dealing with complex construction situations. By integrating dynamic visual data and static bridge construction knowledge, and realizing real-time human-computer interaction through an intelligent question-and-answer system, this patent significantly improves the intelligent level of bridge construction management. In addition, the three-dimensional expression function of the system can visualize complex construction data, provide intuitive construction progress and structural quality information to engineers, and help them better understand the construction progress and make decision analysis.

Claims

1. An intelligent auxiliary decision-making method for bridge construction progress based on multimodal visual language model, It is characterized in that The steps include: Step 1: Obtain and pre-process the bridge construction progress data, including engineering data, construction data, monitoring images and on-site data; Step 2: construct a multimodal bridge construction decision analysis model based on image-text information fusion, and train it with the bridge construction progress data obtained through preprocessing; Step 3: Preprocess the real-time acquired bridge construction progress data and input it into the trained image-text information fusion multimodal bridge construction decision analysis model to integrate the static scene data and data dynamic analysis of the bridge construction site, and construct a unified fusion expression model for real-time feedback of construction progress and quality based on the instructions of multi-level user task requirements, that is, to obtain a three-dimensional model of the bridge construction progress scene.

2. According to claim 1, a method for intelligent auxiliary decision-making of bridge construction progress based on a multimodal visual language model, It is characterized in that The specific steps of step 2 are: Step 2.1, construct a bridge construction progress target monitoring model based on the improved YOLO-Bridge3D and a three-dimensional information decoding predictor and train it through the pre-processed bridge construction progress data to obtain bridge construction structure detection; Step 2.2, construct a multimodal private domain VectorStore structure database and a bridge construction expert LLM model based on dynamic monitoring data and private knowledge information, and train the detected bridge construction structure to obtain knowledge-guided construction status analysis, and finally obtain a trained image-text information fusion multimodal bridge construction decision analysis model.

3. According to claim 1, a method for intelligent auxiliary decision-making of bridge construction progress based on a multimodal visual language model, It is characterized in that The improved YOLO-Bridge3D bridge construction progress target monitoring model in step 2.1 is based on the Yolov5s model, adding a CBAM channel attention mechanism module to its backbone network, and adding a DiceLoss function to the head network to obtain boundary information, including the structural type of the bridge, spatial morphological constraints and boundary discrete information.

4. According to claim 3, a method for intelligent auxiliary decision-making of bridge construction progress based on a multimodal visual language model, It is characterized in that The improved YOLO-Bridge3D bridge construction progress target monitoring model adds CBAM channel attention mechanism modules between the first Conv2D layer and the second Conv2D layer, between the second Conv2D layer and the third Conv2D layer, between the third Conv2D layer and the fourth Conv2D layer, and after the SPPF layer in the backbone network of the Yolov5s model; The formula of the Dice Loss function is: Among them, P i and G i They represent the predicted result and the true label value at the i-th pixel respectively, and N represents the total number of pixels.

5. According to claim 4, a method for intelligent auxiliary decision-making of bridge construction progress based on a multimodal visual language model, It is characterized in that The CBAM channel attention mechanism module includes a channel attention mechanism module that receives the input features of the previous layer output, and a spatial attention mechanism module that receives the result of channel multiplication of the output of the channel attention mechanism module and the input features. The result of dimension multiplication of the input and output of the spatial attention mechanism module is the output of the CBAM channel attention mechanism module; The channel attention mechanism module includes a global maximum pooling layer and a global average pooling layer that receive the input features of the previous layer output in parallel, a multi-layer perceptron connected to the global maximum pooling layer and the global average pooling layer, and a Sigmoid activation function layer that generates a channel attention map based on the output summation result of the multi-layer perceptron. The channel attention map is the output of the channel attention mechanism module; The spatial attention mechanism module includes a global maximum pooling layer, a global average pooling layer, a 7*7 convolution layer and a Sigmoid activation function layer, which sequentially receive the output of the channel attention mechanism module and the multiplication of the input feature. The output of the Sigmoid activation function layer is a spatial attention map, that is, the output of the spatial attention mechanism module, wherein the inputs of the global maximum pooling layer and the global average pooling layer are channel-concatenated and used as the input of the 7*7 convolution layer.

6. According to claim 2, a method for intelligent auxiliary decision-making of bridge construction progress based on a multimodal visual language model, It is characterized in that The specific implementation steps of the three-dimensional information decoding predictor are: Step 2.11: First, use the Visvalingam-Whyatt algorithm to simplify the feature information of the boundary discrete information output by the improved YOLO-Bridge3D bridge construction progress target monitoring model, and match the simplified feature information. The specific steps are as follows: First, calculate the area of ​​each triangle formed by three consecutive points. The calculation formula is: Among them, (x 1 ,y 1 ), (x 2 ,y 2 ), (x 3 ,y 3 ) are the coordinates of the three vertices of the triangle, and the point sequence P 1 ,P 2 ,P 3 ...P m , for each point P l , l takes the value from 2 to m-1, then the calculation is done by P l-1 ,P l ,P l+1 The area of ​​the triangle formed; Then, find the triangle with the smallest area from all the calculated triangles and remove the middle point P of the smallest triangle. l , and remove P l After the point is found, update the areas of the two triangles adjacent to the smallest triangle; Next, the previous step is repeated, and the middle point of the triangle with the smallest area is removed in each iteration until the preset simplification threshold is reached, and the simplified feature information is obtained; Finally, based on the simplified feature information, the FLANN matcher is used to quickly find the nearest neighbor actual boundary discrete points in the high-dimensional space, and the actual boundary discrete points are matched and calibrated with the expected reference points of the theoretical model; Step 2.12, using GPS geographic information to match the bridge construction progress data of the bridge construction site, and clarifying the design form of the bridge construction site based on the construction technology used at the bridge construction site and the structural type of the bridge, and then using the spatial form constraints of the bridge to establish the bridge construction site constraint conditions; Step 2.13, based on the constraints and matching results, the photographic scale conversion coefficient is calculated using the perspective projection principle from three-dimensional space to two-dimensional plane, and the size information of the changing dimension is deduced from the projection of a fixed two-dimensional plane in the real three-dimensional space on the image plane, that is, the three-dimensional structural size information of the bridge construction progress is decoded.

7. According to claim 2, a method for intelligent auxiliary decision-making of bridge construction progress based on a multimodal visual language model, It is characterized in that The specific steps of constructing the multimodal private domain VectorStore structure database in step 2.2 are: On the basis of dynamic monitoring data and private knowledge information, a non-structured relational database is established by using word vectorization and format conversion methods. That is, after obtaining the number of bridge construction progress vectors that need to be stored and the dimension of each vector, a vector database with nearest neighbor search as the core is established. Then, the inverted file indexing technology is used to find the data point closest to the output result of the three-dimensional information decoding predictor in the relational database. The data points obtained from the query are combined with the information input by the user to form a prompt input template and input into the trained bridge construction expert LLM model for knowledge-guided construction status analysis, including analyzing the stage of bridge construction, judging the time difference between the current construction progress and the theoretical construction progress, giving whether the current construction status is normal, ahead of schedule or delayed, and analyzing and judging the difference between the current construction period and the actual construction period by a few days.

8. According to claim 7, a method for intelligent auxiliary decision-making of bridge construction progress based on a multimodal visual language model, It is characterized in that The specific steps of fine-tuning and improving the language model ChatGLM based on P-tuning in step 2.2 to obtain the bridge construction expert LLM model based on dynamic monitoring data and private knowledge information are as follows: First, construct the prompt and create a learnable virtual token as the prompt template of the model; Then, the virtual token of knowledge and experience is initialized and embedded into the input sequence of the language model ChatGLM, that is, when the language model ChatGLM processes the input, the virtual token is regarded as a part of the sequence, and the virtual token interacts through the self-attention mechanism of the language model ChatGLM to obtain the fine-tuned language model ChatGLM, that is, the bridge construction expert LLM model of dynamic monitoring data and private knowledge information is obtained; wherein, the virtual token of knowledge and experience is initialized and embedded into the input sequence of the language model ChatGLM refers to further introducing a method of continuous prompt embedding on the basis of discrete prompt search to obtain the fine-tuned language model ChatGLM, that is, the bridge construction expert LLM model of dynamic monitoring data and private knowledge information is obtained, and the specific steps are as follows: First, the pre-trained language model ChatGLM is initialized, that is, the language model ChatGLM is initialized to have a specific hidden layer size H and vocabulary size |V|, wherein the hidden layer size H refers to the number of neurons in the hidden layer of the language model ChatGLM or the dimension of the feature vector, and the weight matrix and bias vector of the language model ChatGLM are further initialized according to the hidden layer size H set by the initialization, and the vocabulary size |V| refers to the number of words or subword units in the vocabulary of the language model ChatGLM that contain information in the field of bridge construction; Next, we construct and use the word information provided by the vocabulary |V| to form a template T using discrete prompt tags. The template T organizes the input discrete data into a series of text tags, denoted by T = {[D 0:i ],x,[D (i+1):j ],y,[D (j+1):k ]}, where [D i ] represents a discrete prompt mark, i and j both represent the location of the discrete prompt mark, [D 0:i ] represents a discrete sequence of prompt tokens from index 0 to i, [D (i+1):j ] represents a discrete prompt tag sequence from index i+1 to j, [D (j+1):k ] represents the discrete prompt tag sequence from index j+1 to k, that is, it includes all discrete prompt tags from the h+1th position to the kth position, where the discrete data includes the annotation data x and the label y; Secondly, the discrete data is applied to the template T; By keeping the language model ChatGLM parameters frozen, we use an additional embedding function f(P i ):{h 0 ,...,h j ,e(x),h i+1 ,...,h j ,e(y),h j+1 ,...,h K } Make a mapping template to map the discrete prompt tags and discrete data together to the pre-trained embedding layer to construct a new prompt template T = {[P 0:i ],x,[P (i+1):j ],y,[P (j+1):k ]}, the fine-tuned language model ChatGLM is obtained, that is, the bridge construction expert LLM model with dynamic monitoring data and private knowledge information is obtained, which is used for training with the Prompt input template to obtain {e(D 0 )...e(D j ),e(x 0 ),...,e(x n ),e(D k )}, where e represents the embedding matrix, P i is the i-th consecutive cue embedding, h 0 represents the hidden layer state at the 0th layer position, h j represents the hidden layer state of the jth layer, h i+1 represents the hidden layer state at the i+1th layer position, h K represents the hidden layer state at the Kth layer position, e(x) and e(y) represent the embedded representations of different labeled data x and labels y respectively, [P 0:i ] represents the discrete hint tag embedding of the 0th to ith parts, [P (i+1):j ] represents the discrete hint tag embedding of the i+1th to jth parts, [P (j+1):k ] represent the discrete cue tag embeddings from the j+1th to the kth part, e(D 0 )、e(D j )、e(x 0 )、e(x n )、e(D k ) represent the embedded representations of different discrete prompt tag sequences, respectively. The parameter freezing of the language model ChatGLM includes the freezing of the main weights, that is, the freezing of the weights and bias parameters of the self-attention layer.

9. An intelligent auxiliary decision-making system for bridge construction progress based on a multimodal visual language model, comprising a processor and a memory storing a computer program, Features: The processor executes the computer program to implement the method according to any one of claims 1 to 8.

Images