A construction site safety inspection method based on multimodal large language model

Abstract

The present invention belongs to the technical field related to construction safety inspection, and specifically relates to a construction site safety inspection method based on a multimodal large language model, including: introducing a multimodal large language model, combining with a knowledge base containing safety standards and evaluation rules, and applying the multimodal large language model multiple times, specifically: obtaining an image content description by prompting text from an image and a preset visual perception system; intelligently decomposing the user input query into multiple sub-query tasks by prompting a user input query, an image content description, and a preset task decomposition system; searching the construction safety knowledge base for construction safety management standard data related to each sub-query, analyzing and reasoning the sub-query based on the image content description and combined with the searched standard data, obtaining the reasoning and conclusion of the sub-query; and finally obtaining the overall reasoning and conclusion based on each sub-query data. The present invention can accurately, scientifically, and comprehensively realize construction site safety inspection.

Description

Technical Field

[0001] The present invention belongs to the technical field related to construction safety inspection, and more specifically, relates to a construction site safety inspection method based on a multimodal large language model. Background Art

[0002] Safety management has always been one of the key tasks in the construction industry. However, construction sites face a variety of safety hazards and risks, such as injuries from lifting machinery, collapse of support systems, object strikes, and falls from heights, which pose a serious threat to the life safety of on-site workers and managers. Existing text data such as various safety standards, guidelines, and accident reports provide an important basis for safety risk assessment, but traditional safety inspection and monitoring methods mainly rely on manual work, which not only requires a large amount of human resources, but also the inspection process is limited by human experience and subjective judgment. Once a mistake occurs, it is very likely to affect the efficiency and real-time nature of construction site safety management.

[0003] In recent years, with the continuous development of technologies such as smart sensors, smart mechanical equipment, and computer vision, a large amount of construction site safety data in various formats such as text, images, and videos has been accumulated. Through supervised learning methods such as manual labeling and screening of these safety data, various automated safety inspection and hazard detection methods have gradually been developed and applied, which has improved the efficiency of on-site safety monitoring to a certain extent and reduced the risk of misjudgment caused by human factors. However, the safety hazards at construction sites are dynamic and diverse, which requires various types of automatic safety inspection methods and hazard assessment systems to quickly adapt to new management and control indicators. However, existing automated safety inspection methods require the preparation of a large number of additional labeled data sets and retraining of models, which is time-consuming and labor-intensive and it is difficult to achieve comprehensive coverage of new scenarios in a short period of time.

[0004] In addition, with the continuous improvement of construction safety management standards, new safety standards and evaluation rules need to be continuously introduced. The existing automated safety inspection methods lack in-depth semantic understanding and logical reasoning capabilities, and it is difficult to comprehensively consider the rich knowledge and experience contained in various safety standards, specifications, and accident reports. When faced with a complex and changing construction environment, the existing methods are unable to analyze the multimodal safety data collected on site in real time and comprehensively, and accurately and scientifically give safety risk assessment opinions for the construction site. Summary of the invention

[0005] In response to the above defects or improvement needs of the prior art, the present invention provides a construction site safety inspection method based on a multimodal large language model, the purpose of which is to construct a multimodal large language model, combine a knowledge base containing safety standards and evaluation rules in a single construction site safety inspection task, apply the multimodal large language model multiple times, and finally obtain the comprehensive construction safety diagnostic reasoning and conclusions required by the user, thereby solving the technical problem that the existing methods are unable to analyze the multimodal safety data collected on site in real time and comprehensively, and accurately and scientifically give construction site safety risk assessment opinions.

[0006] To achieve the above object, according to one aspect of the present invention, a construction site safety inspection method based on a multimodal large language model is provided, comprising:

[0007] The user input image and the preset visual perception system prompt text are respectively input into the image representation branch and the text representation branch of the multimodal large language model, and the generative pre-trained transformation model GPT in the multimodal large language model outputs the image content description; wherein the image representation branch is used to represent the image as the image semantic embedding of the construction scene semantic space, and the text representation branch is used to represent the text input as the text semantic embedding of the construction scene semantic space, and the outputs of the two branches are connected to the input of the GPT; the GPT is used to complete the corresponding engineering safety diagnosis task according to the input embedding information and output the result;

[0008] Inputting a user input query, the image content description and a preset task decomposition system prompt text into the text representation branch, and the GPT outputting a plurality of sub-queries corresponding to the user input query;

[0009] Each subquery is input into the construction safety knowledge base, and the search module in the construction safety knowledge base searches the text block in the construction safety management standard data file that is closest to the subquery from the vector database in the construction safety knowledge base; the image content description, each subquery, the text block closest to the subquery, and the preset subtask reasoning system prompt text are input into the text representation branch, and the GPT outputs the reasoning and conclusion of the subquery;

[0010] The image content description, the user input query, the reasoning and conclusion of each sub-query, and the preset comprehensive reasoning task system prompt text are input into the text representation branch, and the GPT outputs the comprehensive reasoning and conclusion for the user input query.

[0011] Furthermore, the multimodal large language model is constructed in the following way:

[0012] Constructing an instruction sample set, each instruction sample includes an image, an image content annotation, and a safety diagnosis annotation; the image content annotation is a content description of various safety management elements, and the safety diagnosis annotation is a causal reasoning about safety management elements with potential risks;

[0013] Build a multimodal large language model architecture, including a generative pre-trained transformation model GPT, an image representation branch, and a text representation branch; the image representation branch is used to represent the image in the general scene semantic space through the visual encoder, and convert the representation result to the construction scene semantic space through the mapping unit to obtain the image semantic embedding; the text representation branch is used to represent the text input as a text semantic embedding; both the image semantic embedding and the text semantic embedding are used as the input of the generative pre-trained transformation model GPT;

[0014] The instruction sample set is adopted, the input of the text representation branch is left blank and the parameters of the visual encoder are fixed, and the parameters of the generative pre-trained transformation model GPT and the mapping unit are tuned.

[0015] Furthermore, the search module retrieves the text block that is closest to the query-related instruction through a hybrid search method of keyword search and semantic search; wherein, a sparse vector search method is used to perform keyword search; and a dense vector search method is used to perform semantic search.

[0016] Furthermore, the construction safety knowledge base also includes a knowledge processing unit, which is used to perform text segmentation on each document in the construction safety management standard data file by semantic segmentation; convert each obtained text block into a high-dimensional vector by text embedding;

[0017] Then the vector database stores the high-dimensional vectors processed by the knowledge processing unit and their corresponding text blocks;

[0018] The high-dimensional vector is used to measure semantic similarity when executing the semantic retrieval to search for a text block that is closest to the query instruction.

[0019] Furthermore, the construction safety management standard data file includes safety regulations, historical safety accident reports and comprehensive safety guidelines.

[0020] Furthermore, the mapping unit is a multi-layer perceptron network.

[0021] Furthermore, the visual perception system prompt text is:

[0022] “You are a construction management assistant specializing in describing construction images. Please describe in detail the construction activities, tools, and structures observed in the images. Note: When describing the images, you can ignore the user query and focus on the image content.”

[0023] Furthermore, the task decomposition system prompts the text prompt as follows:

[0024] “Given a user query and image description, output a list of relevant subqueries in json markdown format that can help answer the user query; these subqueries will be used to search relevant engineering safety code documents to obtain regulations and recommendations on safety practices;

[0025] The following are guidelines to follow when building subqueries:

[0026] 1. Analyze the provided picture descriptions to identify specific construction activities, tools, or structures;

[0027] 2. Generate a subquery that requires the document to provide correct security procedures;

[0028] 3. Ensure that the sub-questions help to find relevant regulations on correct safety practices;

[0029] 4. Make sure the sub-questions are relevant to the input query and image description”;

[0030] The user query is a query input by a user, and the picture description is a description of the image content.

[0031] According to another aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0032] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the steps of the method described above.

[0033] In general, compared with the prior art, the above technical solution conceived by the present invention has the following beneficial effects:

[0034] 1. The present invention provides a construction site safety inspection method based on a multimodal large language model. The method introduces a multimodal large prediction model. The multimodality is reflected in the fact that it has an image representation branch and a text representation branch. These two branches can receive and process two modal data, image and text query information, to convert the input data into a data form that can be recognized by the generative pre-trained transformation model GPT, so that the generative pre-trained transformation model GPT performs related task analysis. Through this multimodal technology, the versatility and accuracy of the large language model in safety inspections at construction sites are enhanced. In addition, by integrating the multimodal large language model with the construction safety knowledge base of the field, the ability of the multimodal large language model in performing accurate and context-related safety analysis is improved. Specifically: the multimodal large language model is applied multiple times to imitate the thinking and judgment of the safety inspector. First, the image and the preset visual perception system prompt the text to obtain the image content description; secondly, the user input query is intelligently decomposed into multiple sub-query tasks by the user input query, image content description and preset task decomposition system prompt; further, for each sub-query, first search the construction safety knowledge base for relevant construction safety management standard data, and then analyze and reason each sub-query based on the image content description and the searched construction safety management standard data to obtain the reasoning and conclusion of the sub-query; finally, based on the above data, the overall reasoning and conclusion are analyzed. This way of using the multimodal large language model can provide comprehensive construction site safety analysis, ensure the accuracy of safety inspections in complex construction environments, and meet the key needs of the construction industry for highly robust and adaptable automatic construction safety monitoring and inspection methods.

[0035] 2. The present invention also proposes a method for constructing a multimodal large language model, which fully considers the existing construction safety management experience such as accident investigation reports as domain knowledge, and fine-tunes the multimodal large language model in the form of an instruction sample set to enhance the image and text reasoning capabilities of the large language model.

[0036] 3. The present invention also proposes to perform text block search through a hybrid retrieval method of keyword retrieval and semantic retrieval, which can simultaneously consider the matching degree at the lexical level and the semantic matching degree implied by the lexical level, thereby achieving a more comprehensive matching degree calculation.

[0037] 4. The present invention also proposes a construction safety management standard data file including field-specific information such as safety regulations, historical safety accident reports and comprehensive safety guidelines, so that the knowledge base can include a wide range of engineering safety regulations, best practices and related cases. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1It is a flowchart of a construction site safety inspection method based on a multimodal large language model provided by an embodiment of the present invention.

[0039] Figure 2 It is a schematic diagram of a task decomposition mechanism for user input queries in the automatic security check provided by an embodiment of the present invention.

[0040] Figure 3 It is a flowchart for implementing comprehensive security assessment reasoning based on each sub-query and its corresponding text block provided by an embodiment of the present invention.

[0041] Figure 4 It is a diagram of the architecture of a multimodal large language model provided by an embodiment of the present invention.

[0042] Figure 5 It is a schematic diagram of the text block search process provided by an embodiment of the present invention.

[0043] Figure 6 It is a user input image in an example provided by an embodiment of the present invention.

[0044] Figure 7 It is a schematic diagram of the overall framework corresponding to an implementation method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0046] Embodiment 1

[0047] A construction site safety inspection method based on a multimodal large language model, such as Figure 1 As shown, including:

[0048] The user input image and the preset visual perception system prompt text are respectively input into the image representation branch and the text representation branch of the multimodal large language model, and the generative pre-trained transformation model GPT in the multimodal large language model outputs the image content description; wherein, the image representation branch is used to represent the image as the image semantic embedding of the construction scene semantic space, and the text representation branch is used to represent the text input as the text semantic embedding of the construction scene semantic space, and the outputs of the two branches are connected to the input of GPT; GPT is used to complete the corresponding engineering safety diagnosis task according to the input embedding information and output the result;

[0049] Input the user input query, image content description and preset task decomposition system prompt text into the text representation branch, and GPT outputs multiple sub-queries corresponding to the user input query;

[0050] Each subquery is input into the construction safety knowledge base, and the search module in the construction safety knowledge base searches the text block in the construction safety management standard data file that is closest to the subquery from the vector database in the construction safety knowledge base; the image content description, each subquery, the text block closest to the subquery, and the preset subtask reasoning system prompt text are input into the text representation branch, and the GPT outputs the reasoning and conclusion of the subquery;

[0051] The image content description, user input query, reasoning and conclusion of each sub-query, and the preset comprehensive reasoning task system prompt text input into the text representation branch, and GPT outputs the comprehensive reasoning and conclusion for the user input query.

[0052] With the continuous improvement of construction safety management standards, new safety standards and evaluation rules need to be continuously introduced. The existing automated safety inspection methods lack in-depth semantic understanding and logical reasoning capabilities, and it is difficult to comprehensively consider the rich knowledge and experience contained in various safety standards, specifications and accident reports. When facing a complex and changeable construction environment, the existing methods cannot analyze the multimodal safety data collected on site in real time and comprehensively, and accurately and scientifically give safety risk assessment opinions on the construction site. In view of the above defects or improvement needs of the prior art, this embodiment provides a construction site safety inspection method based on a multimodal large language model, the purpose of which is to construct a multimodal large language model, combine the knowledge base containing safety standards and evaluation rules in a single construction site safety inspection task, and apply the multimodal large language model multiple times, and finally obtain the comprehensive construction safety diagnostic reasoning and conclusion required by the user, thereby improving the real-time and accuracy of the construction site safety risk assessment, thereby solving the technical problem that the existing methods cannot analyze the multimodal safety data collected on site in real time and comprehensively, and accurately and scientifically give safety risk assessment opinions on the construction site.

[0053] Specifically, it receives the real-time image of the building site safety inspection input by the user, and instructs the multimodal large language model to analyze the scene described by the image and generate a scene description; according to the preset task decomposition system prompt text, user input query and image content description, based on chain thinking, the large language model is prompted to simulate the thinking process of the safety management expert and decompose the task, such as Figure 2 As shown; then, for each sub-query task, the knowledge base is called to accurately retrieve the domain knowledge required to answer the user's query; based on the retrieved domain knowledge, the multimodal large language model generates a complete and comprehensive text description of the security risk information contained in the input image and generates a comprehensive security assessment opinion, such as Figure 3 shown.

[0054] The method of this embodiment integrates the construction site description, knowledge search and reasoning process into a model structure, and decouples the steps of safety knowledge reasoning and visual perception, thereby improving the generalization (i.e., judgment of different unsafe scenarios) and adaptability (understanding of different regulations and knowledge environments) capabilities of the large language model when performing safety inspection tasks.

[0055] It should be noted that the method of this embodiment involves multiple preset prompt texts, such as visual perception system prompt text and task decomposition system prompt text. The visual perception system prompt text means a system prompt that instructs the large language model to perform visual perception cognition, and the task decomposition system prompt text means a system prompt that instructs the large language model to perform task decomposition. In addition, the generative pre-trained transformation model GPT is used to complete the corresponding engineering safety diagnosis task and output the result according to the input embedded information. This function of GPT is trained during the multimodal large language model construction stage.

[0056] As a preferred implementation, the multimodal large language model is constructed in the following way:

[0057] Construct a set of instruction samples, each of which includes an image, image content annotations, and safety diagnosis annotations; image content annotations are content descriptions of various safety management elements, and safety diagnosis annotations are causal reasoning about safety management elements with potential risks; this instruction sample set is actually used to guide the large language model to conduct safety inspections on construction sites;

[0058] Build a multimodal large language model architecture, including a generative pre-trained transformation model GPT, an image representation branch, and a text representation branch; the image representation branch is used to represent the image in the general scene semantic space through the visual encoder, and convert the representation result to the construction scene semantic space through the mapping unit to obtain the image semantic embedding; the text representation branch is used to represent the text input as a text semantic embedding; both the image semantic embedding and the text semantic embedding are used as the input of the generative pre-trained transformation model GPT; the architecture is as follows Figure 4 As shown;

[0059] Using the instruction sample set, the input of the text representation branch is left blank and the parameters of the visual encoder are fixed, and the parameters of the generative pre-trained conversion model GPT and the mapping unit are tuned.

[0060] This preferred method modifies the general generative pre-trained conversion model GPT architecture, adds a pre-trained visual encoder, and introduces a mapping unit to solve the embedding alignment problem between the image and the large language model, so that the multimodal large language model can better understand and analyze the visual information of the construction site. Furthermore, based on the instruction sample set, some parameters of the multimodal large language model are tuned to ensure that the multimodal large language model generates text that meets the needs of the construction safety field while maintaining extensive language knowledge, thereby efficiently and accurately realizing the applicability of the large model to safety diagnosis in the construction safety field.

[0061] It should be noted that the training samples of the multimodal large language model contain annotations (i.e., labels) related to image content description and annotations (i.e., labels) related to safety diagnosis (i.e., reasoning and conclusion). Based on these two aspects of annotations, the image content description ability and reasoning analysis ability of the large language model are trained and established at the same time. These two functions of the large language model are reflected in the different steps of using the large language model in the safety inspection method of this embodiment. Image content annotation and safety diagnosis annotation can help the large language model understand the safety risk assessment standards, so that the model can imitate the thinking and judgment of safety inspectors.

[0062] In addition, for each instruction sample, only image modality data is used as input, but no text modality data. The reason is that before tuning, the generative pre-trained transformation model GPT is a pre-trained model, and its text processing capability has been built in the pre-training stage. In the tuning stage, the instruction sample set is mainly used to improve the large language model's understanding and processing capabilities of images. Therefore, in the tuning stage, only images can be input into the image representation branch, while the input of the text representation branch is left blank.

[0063] The construction method of this embodiment fully considers the existing construction safety management experience such as accident investigation reports as domain knowledge, and fine-tunes the large language model in the form of instruction sample sets to improve the image and text reasoning capabilities of the large model.

[0064] In actual implementation, when constructing instruction sample sets, important images can be collected from various construction sites, including various safety management elements under different construction conditions, such as workers, construction machinery, building materials, structural components, site environment, etc. Detailed image content descriptions (as image content annotations) are added to the images, such as whether there are safety protection equipment, the characteristics of the structure, and the construction activities being carried out in the scene; detailed teaching annotations (i.e., safety diagnosis annotations) are added to the images to provide context and explain why certain safety management elements in the image are considered unsafe or risky.

[0065] Furthermore, since the multimodal large language model is a new large language model architecture proposed in the embodiment of the present invention, the input format of the large language model is now explained, that is, the instruction sample is standardized and then input into the multimodal large language model. In this embodiment, the image data and annotation standardization processing includes: the model input is X, the model output is Y, v represents the image, q represents the query text entered by the user, content represents the content description of the image, instruction represents the safety diagnosis annotation, and i is the index number of the image and related annotations. An example of specific data preparation can be expressed as:

[0066] user:X v,i ,X q <stop>\n assistant:Y content,i +Y instruction,i <stop>\n

[0067] Among them, X v,i represents the i-th image of the input, X q Indicates user input query, Y content,i represents the image content description of the i-th image, Y instruction,i represents the safety diagnosis annotation of the ith image.

[0068] After standardizing the instruction sample input format, the parameters in the multimodal large language model are adjusted through the gradient descent training method.

[0069] As another preferred implementation, the above-mentioned search module retrieves the text block closest to the query-related instructions through a hybrid retrieval method of keyword retrieval and semantic retrieval; wherein, a sparse vector retrieval method is used to perform keyword retrieval; and a dense vector retrieval method is used to perform semantic retrieval.

[0070] As a further preferred implementation, the construction safety knowledge base also includes a knowledge processing unit, which is used to perform text segmentation on each document in the construction safety management standard data file by semantic segmentation. Specifically, based on the contextual semantic relationship of the text, the text is segmented to avoid interrupting sentences or paragraphs, retain semantic integrity, and determine text breakpoints, thereby decomposing each document into smaller, manageable text blocks; each obtained text block is converted into a high-dimensional vector by text embedding; the vector database stores the high-dimensional vectors and their corresponding text blocks processed by the knowledge processing unit. That is, the knowledge processing unit processes the input construction safety management standard data file by text segmentation and embedding technology, converts the segmented text into a high-dimensional vector representation, captures its semantic meaning, creates a semantic space, brings similar text blocks close to each other, and uses, for example, a general embedding (BGE) model to embed the segmented text to support efficient similarity search and information retrieval; wherein the above-mentioned high-dimensional vector is used to measure semantic similarity when performing semantic retrieval to search for the text block closest to the query instruction.

[0071] The knowledge processing unit can obtain data files such as construction safety management standards, relevant safety implementation measures, safety accident reports, etc. in real time, realize continuous safety data acquisition and knowledge accumulation, enable the model and system to continuously adaptively learn and continuously improve, and continuously improve the ability to identify and assess construction safety risks.

[0072] Keyword retrieval measures the relevance of vocabulary matching to find the text block that is closest to the query-related instructions, that is, using sparse vector retrieval (BM25) to ensure the relevance of vocabulary matching (keyword matching). Semantic retrieval is measured by the cosine similarity of dense vectors, that is, using dense vector retrieval technology to capture the semantic similarity between query and document (semantic matching), such as Figure 5 In the above security inspection method, when the image content description, each sub-query, the text block closest to the sub-query, and the preset sub-task reasoning system prompt text are input into the text representation branch, one input method is to splice and input the text blocks of each sub-query obtained by the two search methods respectively.

[0073] In one example of this embodiment, BM25 is selected as the sparse vector retrieval method. q i It is i query words, f(q i , d) yes q i In the documentation q The frequency in | d |Is the document d is the length of the document in words, avgdI is the average length of the documents in the collection, k 1 and b is a free parameter, usually chosen as k 1 ∈[1.2,2.0], b =0.75, IDF( q i )yes q i The anti-document frequency, N is the total number of documents in the collection, n(q i ) is included q i The number of documents, the similarity score calculation method based on BM25 is as follows:

[0074]

[0075]

[0076] In one example of this embodiment, the semantic matching degree is calculated as follows:

[0077]

[0078] In the formula, and Represents two high-dimensional vectors used to calculate semantic matching.

[0079] In this preferred embodiment, the hybrid search method of keyword search and semantic search can simultaneously consider the matching degree at the vocabulary level and the semantic matching degree implied by the vocabulary, thereby achieving a more comprehensive matching degree calculation.

[0080] As another preferred implementation, the above-mentioned construction safety management standard data file includes field-specific information such as safety regulations, historical safety accident reports and comprehensive safety guidelines, so that the knowledge base can include a wide range of engineering safety regulations, best practices and related cases.

[0081] As another preferred implementation, the mapping unit is a multi-layer perceptron network. In fact, by using the security inspection method, the multi-layer perceptron network here is a perceptron network with less than 5 layers, which can be used to solve the technical problem of the present invention while ensuring the calculation rate.

[0082] As another preferred implementation, the visual perception system prompt text is:

[0083] "You are a construction management assistant specializing in describing construction imagery; please describe in detail the construction activities, tools, and structures observed in the image. Note: When describing the image, you can ignore the user query and focus on the image content."

[0084] It should be noted that the prompt text is an optimized text form that is used for reasoning analysis in a multimodal large language model with relatively accurate results. If the words or sentence forms are changed, the output results of GPT will be affected. For example, if the period "." in the above text is changed to a semicolon ";", the obtained embedding vector will change after entering the text embedding branch of the large language model, thus affecting the reasoning output of GPT. However, it should be noted that in this text form, changing the period "." to a semicolon ";" will not have much effect, but if other parts of the text are changed, the impact may be relatively large.

[0085] As another preferred implementation, the above-mentioned task decomposition system prompt text prompt is:

[0086] "Given a user query and an image description, output a list of relevant subqueries in json markdown format that can be composed to help answer the user query. These subqueries will be used to search relevant engineering safety specification documents for regulations and recommendations on safety practices.

[0087] The following are guidelines to follow when building subqueries:

[0088] 1. Analyze the provided picture descriptions to identify specific construction activities, tools, or structures.

[0089] 2. Generate subqueries that require the document to provide correct security procedures.

[0090] 3. Ensure that the sub-questions help to find relevant regulations on correct safety practices.

[0091] 4. Make sure the sub-questions are relevant to the input query and image description”;

[0092] The user query is a query input by a user, and the picture description is a description of the image content.

[0093] Similar to the visual perception system prompt text mentioned above, this prompt text is also an optimized text form that is more accurate for reasoning and analysis by a multimodal large language model. If the words or sentence forms are changed, the output results of GPT will be affected. For example, if the period "." in the above text is changed to a semicolon ";", the obtained embedding vector will change after entering the text embedding branch of the large language model, thereby affecting the reasoning output of GPT. However, it should be noted that in this text form, changing the period "." to a semicolon ";" will not have much effect, but if you change to something else, the impact may be relatively large.

[0094] In order to better illustrate the present invention, an example is now given.

[0095] User input image such as Figure 6 As shown, the text entered by the user for the query is: "Is there any potential safety hazard in the diagram?". The method of the present invention is adopted, and the specific steps are as follows:

[0096] S1. Input the image into the user ( Figure 6 ) and the preset visual perception system prompt text ("You are a full-time construction management assistant who specializes in describing building images; please describe in detail the construction activities, tools, and structures observed in the image. Note: When describing the image, you can ignore the user query and focus on the image content"), which are respectively input into the image representation branch and text representation branch of the multimodal large language model, and the generative pre-trained conversion model GPT in the multimodal large language model outputs the image content description as follows:

[0097] "The image depicts a construction site with several workers and some heavy machinery. In the foreground, there are two men wearing high-visibility vests, hard hats, and safety goggles; they appear to be performing manual labor, possibly related to excavation or ground preparation. One of the workers is using a shovel, while the other is holding a tool that resembles a pickaxe.

[0098] In the background, there is a large excavator with its bucket extended into the ground, suggesting that it may be digging or moving material. The sky is partly cloudy, suggesting that the weather is fine for outdoor work.

[0099] The construction site itself appears to be in its early stages, as no significant buildings are visible. Workers are focused on their work and machinery is neatly arranged, typical of an active construction site. The ground is uneven, indicating that earthworks have been or are in progress.

[0100] Overall, this image captures a moment in the development of a new infrastructure project, highlighting the teamwork and coordination required to safely and efficiently execute such large-scale construction activities."

[0101] S2, the user input query ("Is there a safety hazard in the image?"), the image content description (i.e., the output of S1), and the preset task decomposition system prompt text ("Given the user query and image description, output a list of related subqueries in jsonmarkdown format, which can help answer the user query after composition. These subqueries will be used to search relevant engineering safety specification documents to obtain regulations and recommendations on safety practices. The following are guidelines to be followed when generating subqueries: 1. Analyze the provided image description to identify specific construction activities, tools, or structures. 2. Generate subqueries that require documents to provide correct safety procedures. 3. Ensure that the subquestions help find relevant regulations on correct safety practices. 4. Ensure that the subquestions are related to the input query and image description") are input into the text representation branch, and GPT outputs multiple subqueries corresponding to the user input query, as follows:

[0102] Subquery 1: "Are workers on top of the pile properly secured to prevent the risk of falling?"

[0103] Subquery 2: "Are there any potential dangers in operating an excavator in close proximity to other workers?"

[0104] Subquery 3: "Is there a risk of ground instability or collapse at the construction site that could endanger workers working at heights?"

[0105] Subquery 4: "Whether appropriate safety procedures regarding personal protective equipment (PPE) and heavy machinery operating procedures were followed at the construction site."

[0106] S3, input each subquery into the construction safety knowledge base, and the search module in the construction safety knowledge base searches the text block in the construction safety management standard data file that is closest to the subquery from the vector database in the construction safety knowledge base; input the image content description (output of S1), each subquery (output of S2), the text block closest to the subquery, and the preset subtask reasoning system prompt text into the text representation branch, and GPT outputs the reasoning and conclusion of the subquery:

[0107] Reasoning and conclusion of subquery 1: "Yes, operating an excavator in close proximity to other workers is potentially dangerous"

[0108] Reasoning and conclusion of subquery 2: "Yes, there is a risk of unstable or collapsed ground at the construction site, which may endanger workers working at heights"

[0109] Reasoning and conclusion of sub-query 3: "The workers on top of the pile did not take appropriate safety measures to prevent the risk of falling"

[0110] Reasoning and conclusion of sub-query 4: "The heavy machinery at the construction site followed appropriate safety regulations in terms of personal protective equipment (PPE) and operating procedures."

[0111] S4, the image content description (output of S1), the user input query ("Is there a safety hazard in the image?"), the reasoning and conclusion of each subquery (output of S3), and the preset comprehensive reasoning task system prompt text input text representation branch, and GPT outputs the comprehensive reasoning and conclusion for the user input query:

[0112] "The image shows a busy construction site with several workers wearing hard hats and high-visibility vests performing various tasks including excavation, material handling, and site preparation. An excavator operator appears to be digging or moving materials above ground, while another worker stands on a pile of dirt or rubble, possibly supervising operations or performing some task associated with construction activity.

[0113] The presence of heavy machinery such as excavators and cranes indicates that this is a large-scale project involving earthworks and other complex tasks. The ground appears uneven, with obvious trenches and piles of earth, which may pose risks to workers due to potential instability or collapse. In particular, workers on top of the piles face the risk of falling from height and need to take appropriate safety measures such as guardrails, safety belts or other fall protection systems.

[0114] The image also shows the need for clear communication and coordination between workers operating heavy machinery and those working near it. To reduce the risks associated with operating excavators and cranes, all personnel must follow established safety protocols, including maintaining a safe distance from mobile equipment, using monitors when necessary, and following proper signaling procedures.

[0115] In summary, this construction site presents several potential hazards associated with heavy machinery operation, earthmoving activities, and working at height positions. To ensure the safety of all workers on site, management must strictly adhere to OSHA guidelines such as proper use of personal protective equipment, regular safety inspections, hazard identification and mitigation measures, and a comprehensive training program focused on equipment operation, fall protection, and emergency response.”

[0116] A framework diagram of an implementation method, in general, as Figure 7 shown.

[0117] Embodiment 2

[0118] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0119] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory may be used to store computer programs and / or modules, and the processor may perform various functions of the electronic device by running or executing computer programs and / or modules stored in the memory, and calling data stored in the memory.

[0120] The relevant technical solutions are the same as above and will not be described in detail here.

[0121] Embodiment 3

[0122] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0123] Specifically, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0124] The relevant technical solutions are the same as above and will not be described in detail here.

[0125] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.< / stop> < / stop>

Claims

1. A construction site safety inspection method based on a multimodal large language model, characterized in that: include: The user input image and the preset visual perception system prompt text are respectively input into the image representation branch and the text representation branch of the multimodal large language model, and the generative pre-trained transformation model GPT in the multimodal large language model outputs the image content description; wherein the image representation branch is used to represent the image as the image semantic embedding of the construction scene semantic space, and the text representation branch is used to represent the text input as the text semantic embedding of the construction scene semantic space, and the outputs of the two branches are connected to the input of the GPT; the GPT is used to complete the corresponding engineering safety diagnosis task according to the input embedding information and output the result; Inputting a user input query, the image content description and a preset task decomposition system prompt text into the text representation branch, and the GPT outputting a plurality of sub-queries corresponding to the user input query; Each subquery is input into the construction safety knowledge base, and the search module in the construction safety knowledge base searches the text block in the construction safety management standard data file that is closest to the subquery from the vector database in the construction safety knowledge base; the image content description, each subquery, the text block closest to the subquery, and the preset subtask reasoning system prompt text are input into the text representation branch, and the GPT outputs the reasoning and conclusion of the subquery; The image content description, the user input query, the reasoning and conclusion of each sub-query, and the preset comprehensive reasoning task system prompt text are input into the text representation branch, and the GPT outputs the comprehensive reasoning and conclusion for the user input query.

2. The construction site safety inspection method according to claim 1, characterized in that: The multimodal large language model is constructed in the following way: Constructing an instruction sample set, each instruction sample includes an image, an image content annotation, and a safety diagnosis annotation; the image content annotation is a content description of various safety management elements, and the safety diagnosis annotation is a causal reasoning about safety management elements with potential risks; Build a multimodal large language model architecture, including a generative pre-trained transformation model GPT, an image representation branch, and a text representation branch; the image representation branch is used to represent the image in the general scene semantic space through the visual encoder, and convert the representation result to the construction scene semantic space through the mapping unit to obtain the image semantic embedding; the text representation branch is used to represent the text input as a text semantic embedding; both the image semantic embedding and the text semantic embedding are used as the input of the generative pre-trained transformation model GPT; The instruction sample set is adopted, the input of the text representation branch is left blank and the parameters of the visual encoder are fixed, and the parameters of the generative pre-trained transformation model GPT and the mapping unit are tuned.

3. The construction site safety inspection method according to claim 1, characterized in that: The search module retrieves the text block that is closest to the query-related instruction through a hybrid search method of keyword search and semantic search; wherein, a sparse vector search method is used to perform keyword search; and a dense vector search method is used to perform semantic search.

4. The construction site safety inspection method according to claim 3, characterized in that: The construction safety knowledge base also includes a knowledge processing unit for performing text segmentation on each document in the construction safety management standard data file by means of semantic segmentation; converting each obtained text block into a high-dimensional vector by means of text embedding; Then the vector database stores the high-dimensional vectors processed by the knowledge processing unit and their corresponding text blocks; The high-dimensional vector is used to measure semantic similarity when executing the semantic retrieval to search for a text block that is closest to the query instruction.

5. The construction site safety inspection method according to claim 1, characterized in that: The construction safety management standard data files include safety regulations, historical safety accident reports and comprehensive safety guidelines.

6. The construction site safety inspection method according to claim 2, characterized in that: The mapping unit is a multi-layer perceptron network.

7. The construction site safety inspection method according to claim 1, characterized in that: The visual perception system prompt text is: "You are a construction management assistant specializing in describing construction images. Please describe in detail the construction activities, tools, and structures observed in the images. Note: When describing the images, you can ignore the user query and focus on the image content." 8. The construction site safety inspection method according to claim 1, characterized in that: The task decomposition system prompt text is: "Given a user query and an image description, output a list of related subqueries in json markdown format that can help answer the user query; these subqueries will be used to search relevant engineering safety specification documents for regulations and recommendations on safety practices; The following are guidelines to follow when building subqueries:

1. Analyze the provided picture descriptions to identify specific construction activities, tools, or structures; 2. Generate a subquery that requires the document to provide the correct security procedures; 3. Ensure that the sub-questions help to find relevant regulations on correct safety practices; 4. Make sure the sub-questions are relevant to the input query and image description”; The user query is a query input by a user, and the picture description is a description of the image content.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the steps of the method according to any one of claims 1 to 8.

Images