Engineering field multi-mode large model construction safety capability assessment method and related equipment

Abstract

The invention discloses an engineering field multi-mode large model construction safety capability assessment method and related equipment. The engineering field multi-modal large model construction safety capability assessment method comprises the following steps: constructing a multi-dimensional construction safety capability assessment system, establishing a capability assessment framework comprising three dimensions of safety perception, safety description and safety reasoning, and designing a corresponding assessment index for each capability dimension; collecting construction safety multi-scene image data and carrying out image labeling, and constructing a construction safety evaluation data set; a standardized multi-modal construction safety assessment instruction is designed, and different assessment instructions are correspondingly designed for different capability dimensions and construction scenes; selecting a plurality of multi-modal large models, executing a standardized multi-modal construction safety assessment instruction, and performing model assessment on the plurality of multi-modal large models according to a multi-dimensional construction safety capability assessment system; and calculating the performance score of each model in each capability dimension and construction scene.

Description

Technical Field

[0001] The present invention relates to the technical field of construction safety capability assessment, and in particular to a multi-modal large-scale model construction safety capability assessment method and related equipment in the engineering field. Background Art

[0002] In recent years, breakthroughs have been made in the technology of multimodal large language models (MLLMs). Models such as GPT-4o, Claude3.5Sonnet, and Gemini-1.5-Pro ​​have demonstrated powerful capabilities in image and text comprehension, cross-modal reasoning, and knowledge application. These models can simultaneously process information from multiple modalities, such as text and images. Through large-scale pre-training, they acquire a wealth of general knowledge and possess powerful transfer learning and zero-shot learning capabilities. These models provide a new technical approach for addressing the limitations of traditional deep learning methods in construction safety management. Compared to traditional methods, MLLMs do not require specialized training for each type of safety hazard. Instead, they can understand safety risks in construction images based on inherent knowledge and provide assessments and recommendations based on safety regulations and professional knowledge, demonstrating greater versatility and scalability.

[0003] However, the existing MLLMs evaluation benchmarks mainly focus on general capabilities, such as MM-Bench, SEED-Bench, etc. These MLLMs evaluation methods cannot fully reflect the actual performance of MLLMs in the professional field of construction safety. At the same time, construction safety management involves a variety of scenarios (such as foundation pit engineering, scaffolding engineering, edge protection, etc.) and multi-dimensional capabilities (such as safety hazard identification, risk analysis, standard reference, etc.). The existing MLLMs evaluation methods are difficult to fully cover these professional scenarios and capability dimensions. In addition, the professional knowledge in the field of construction safety is highly professional and standardized, so MLLMs are required to have a rich reserve of safety knowledge and strong reasoning ability, but there is currently a lack of effective methods to evaluate the differences in the capabilities of different MLLMs in these aspects, which leads to the lack of scientific basis for engineering construction units to choose models that suit their business needs.

[0004] In summary, there is currently a lack of systematic evaluation methods for the performance of large multimodal models in the field of engineering construction safety. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and related equipment for evaluating the construction safety capabilities of multimodal large models in the engineering field, so as to solve the technical problem that there is currently a lack of a systematic evaluation method for the performance of multimodal large models in the professional field of engineering construction safety.

[0006] First, a method for evaluating the construction safety capabilities of multimodal large models in the engineering field is provided, including: constructing a multidimensional construction safety capability evaluation system, establishing a capability evaluation framework including three dimensions of safety perception, safety description, and safety reasoning, and designing corresponding evaluation indicators for each capability dimension; collecting construction safety multi-scene image data and annotating the images to construct a construction safety evaluation data set; designing standardized multimodal construction safety evaluation instructions, and designing different evaluation instructions for different capability dimensions and construction scenarios; selecting multiple multimodal large models, executing standardized multimodal construction safety evaluation instructions, and performing model evaluation on the multiple multimodal large models according to the multidimensional construction safety capability evaluation system; and calculating the performance score of each model in each capability dimension and construction scenario.

[0007] Secondly, a multimodal large-scale model construction safety capability assessment device in the engineering field is provided, including: a capability assessment system module, which is used to construct a multi-dimensional construction safety capability assessment system, establish a capability assessment framework including three dimensions of safety perception, safety description and safety reasoning, and design corresponding assessment indicators for each capability dimension; an assessment data set construction module, which is used to collect construction safety multi-scene image data and perform image annotation to construct a construction safety assessment data set; an assessment instruction design module, which is used to design standardized multimodal construction safety assessment instructions, and design different assessment instructions corresponding to different capability dimensions and construction scenarios; a model evaluation execution module, which is used to select multiple multimodal large models, execute standardized multimodal construction safety assessment instructions and perform model evaluation on the multiple multimodal large models according to the multi-dimensional construction safety capability assessment system; an evaluation score calculation module, which is used to calculate the performance score of each model in each capability dimension and construction scenario.

[0008] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for assessing the construction safety capability of a multimodal large model in an engineering field when executing the computer program.

[0009] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for assessing the construction safety capability of a multimodal large model in the engineering field are implemented.

[0010] The above-mentioned multimodal large-scale model construction safety capability assessment method, device, computer equipment and computer-readable storage medium in the engineering field, and the constructed construction safety assessment data set cover a variety of construction scenarios such as foundation pit engineering, scaffolding engineering, and edge protection, and have wide applicability. A multidimensional construction safety capability assessment system including three capability dimensions of safety perception, safety description and safety reasoning is constructed. It can comprehensively measure the capabilities of the model from the three dimensions of safety perception, safety description and safety reasoning, and objectively reflect the actual performance of the model in the field of construction safety, so as to accurately identify the advantages and disadvantages of the model in different scenarios and capabilities, and provide a scientific basis for the selection and optimization of multimodal large models in the engineering construction field. It can not only help engineering construction units select the most suitable multimodal large model, but also provide improvement directions for model developers, and promote the application and development of multimodal artificial intelligence technology in construction safety management. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 1 is a flow chart of a method for evaluating construction safety capability of a multi-modal large-scale model in an engineering field according to an embodiment of the present invention;

[0012] Figure 2 1 is a schematic structural diagram of a device for evaluating construction safety capability of a multi-modal large-scale model in an engineering field according to an embodiment of the present invention;

[0013] Figure 3 It is a structural diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0014] In order to enable those skilled in the art to more clearly understand the objectives, technical solutions and advantages of the present invention, the present invention is further described below with reference to the accompanying drawings and embodiments.

[0015] The embodiments of the present application provide a method, apparatus, computer device, and computer-readable storage medium for assessing the construction safety capabilities of a multimodal large-scale model in the engineering field. The method and apparatus for assessing the construction safety capabilities of a multimodal large-scale model in the engineering field can be applied to a server. The present invention is described in detail below using specific embodiments. The following embodiments and features thereof may be combined unless otherwise specified.

[0016] Figure 1 A flow chart of a method for evaluating the construction safety capability of a multi-modal large-scale model in an engineering field according to an embodiment of the present invention is provided. Figure 1 As shown, the engineering field multimodal large model construction safety capability assessment method includes steps S10 to S50:

[0017] S10: Build a multi-dimensional construction safety capability assessment system, establish a capability assessment framework including three dimensions: safety perception, safety description, and safety reasoning, and design corresponding assessment indicators for each capability dimension.

[0018] Construction safety management involves multi-dimensional capabilities (such as safety hazard identification, risk analysis, standard reference, etc.). In order to comprehensively measure the capabilities of multimodal large models, the present invention constructs a multi-dimensional construction safety capability assessment system, establishes a capability assessment framework including three dimensions of safety perception, safety description and safety reasoning, and designs corresponding evaluation indicators for each capability dimension.

[0019] Among them, for safety perception ability, the designed evaluation indicators include hazard object perception and hazard relationship identification; for safety description ability, the designed evaluation indicators include hazard judgment and hazard description; for safety reasoning ability, the designed evaluation indicators include risk analysis, accident consequences, standard index, standard requirements and rectification measures.

[0020] S20: Collect and annotate construction safety multi-scene image data to build a construction safety assessment dataset.

[0021] Construction safety management involves a variety of scenarios. In order to accurately evaluate the capabilities of multimodal large models in various construction scenarios, this paper constructs a construction safety assessment dataset by collecting construction safety multi-scene image data and performing image annotation.

[0022] The collected construction safety multi-scene image data includes a variety of construction scene images including foundation pit engineering, scaffolding engineering, edge protection, unsafe personnel behavior and civilized construction. When the construction safety multi-scene image data is annotated, the image annotation content includes:

[0023] (1) Hazard judgment: Determine whether there are safety hazards in the image, and the judgment result is "yes" or "no".

[0024] (2) Hazard description: Describe the hazards in the image in detail. For example: “Oil drums are randomly stacked, not centrally stored or recycled in a timely manner”, “The flammable material storage area lacks fire-fighting equipment”.

[0025] (3) Hazard objects: Describe the main safety hazard objects in the image, such as oil drums, flammable material piles, acetylene cylinders, etc.

[0026] (4) Hidden danger relationship: Describe the relationship between hidden danger objects and the safety problems they may cause. For example: the distance between the oil drum and the fire source is too close.

[0027] (5) Risk Analysis: Analyze in detail the potential safety risks that the hidden danger may bring. For example, high temperature may cause the oil barrel to explode, causing fire and explosion.

[0028] (6) Accident consequences: Predict the possible consequences of accidents if the hidden danger is not dealt with in a timely manner. Common accident consequences include: falling accidents from heights; accidents caused by objects hitting; accidents caused by injuries caused by lifting machinery; earthwork and foundation pit collapse accidents; accidents caused by injuries caused by construction machinery; electric shock accidents; fire accidents; explosion accidents; poisoning and suffocation accidents, etc.

[0029] (7) Standard index: When citing relevant national standards or industry specifications as a basis, it is necessary to clearly state which specification and which article, as well as the original text of the specification. For example: "Technical Specifications for Fire Safety at Construction Sites of Construction Projects", Article ...

[0030] (8) Standard requirements: List in detail the specific requirements of the safety standards for the hidden danger objects (you can have more than one, and try to list the same knowledge point mentioned in multiple standards). For example: flammable and explosive dangerous goods should be stored in a special warehouse, classified and stored separately, and the warehouse should be well ventilated and clearly marked.

[0031] (9) Corrective measures: Propose effective corrective measures for the hidden danger. For example: store oil drums according to regulations, or contact qualified units to recycle empty drums in a timely manner. It is strictly forbidden to discard or burn them at will.

[0032] S30: Design standardized multimodal construction safety assessment instructions, and design different assessment instructions for different capability dimensions and construction scenarios.

[0033] In step S10, a capability assessment framework is established, encompassing three dimensions: safety perception, safety description, and safety reasoning. Appropriate evaluation indicators are designed for each capability dimension. Consequently, when evaluating a large multimodal model, different evaluation instructions are designed for each evaluation indicator. By executing these instructions, the performance scores of the large multimodal model for each capability dimension and construction scenario are obtained.

[0034] Specifically, the assessment instructions designed for safety perception capabilities include, but are not limited to, hazard object recognition and hazard relationship recognition. The hazard object recognition instruction, when executed, can identify safety hazard objects within an image, such as oil drums, piles of flammable materials, and acetylene cylinders. The hazard relationship recognition instruction, when executed, can identify the relationships between hazard objects and the state attributes of the hazard objects themselves, such as an oil drum too close to a fire source, an improperly fastened safety belt, a helmet not being worn, or a missing safety fence.

[0035] To address safety description capabilities, the designed assessment instructions include, but are not limited to, hazard assessment instructions and hazard description instructions. The hazard assessment instruction, when executed, can determine whether an image contains safety hazards. The hazard description instruction, when executed, can describe the hazards in detail. Examples include: "Oil drums are randomly stacked, not centrally stored or promptly recycled," "A flammable material storage area lacks fire-fighting equipment," and so on.

[0036] In terms of safety reasoning capabilities, the designed evaluation instructions include but are not limited to risk analysis instructions, accident consequence prediction instructions, safety standard reference instructions, standard requirement description instructions, and corrective measures generation instructions. Among them, when the risk analysis instruction is executed, it can analyze in detail the safety risks that the hidden danger may bring. For example: high temperature may cause an oil barrel to explode, causing a fire and explosion. When the accident consequence prediction instruction is executed, it can predict the possible accident consequences if the hidden danger is not dealt with in a timely manner. When the safety standard reference instruction is executed, it can cite relevant national standards or industry specifications as a basis. When the standard requirement description instruction is executed, it can list in detail the specific requirements for hidden danger objects in the safety standard. When the corrective measures generation instruction is executed, it can propose effective corrective measures for the hidden danger.

[0037] S40: Select multiple multimodal large models, execute standardized multimodal construction safety assessment instructions, and perform model assessment on the multiple multimodal large models according to a multi-dimensional construction safety capability assessment system.

[0038] When selecting multiple large multimodal models, you can choose from a number of representative open-source or closed-source models, such as GPT-4o, Claude 3.5Sonnet, Gemini-1.5-Pro, Awaker2.5_VL, Qwen2-VL, InternVL2.5, LLaVA-OneVision, VITA-1.5, Mini-Gemini-34B-HD, MiniCPM-V 2.6, LLaVA1.5-7B, DeepSeek-VL2, LLaVA-NeXT-LLaMA3, and mPLUG DocOwl 1.5. For each selected large multimodal model, execute standardized multimodal construction safety assessment instructions and conduct a model evaluation based on the multi-dimensional construction safety capability assessment system.

[0039] S50: Calculate the performance score of each model in each capability dimension and construction scenario.

[0040] Step S50 further includes S51-S53:

[0041] S51: Use formula (1) to calculate the average Capability Score of each model in each capability dimension, where C is the sample set of one of the capabilities of security perception, security description and security reasoning, and Nc is the number of samples in set C.

[0042]

[0043] S52: Calculate the average Scenario Score of each model in each construction scenario using formula (2), where S is a sample set of one of the scenarios, including foundation pit engineering, scaffolding engineering, edge protection, unsafe behavior of personnel, and civilized construction. Ns is the number of samples in the set S.

[0044]

[0045] S53: Calculate the overall accuracy Safety Score of each model using formula (3), where N is the total number of samples, s i represents the score of sample i,

[0046]

[0047] The multimodal large-scale model construction safety capability assessment method for the engineering field in the embodiment of the present invention constructs a construction safety assessment dataset covering a variety of construction scenarios such as foundation pit engineering, scaffolding engineering, and edge protection, and has wide applicability. It also constructs a multidimensional construction safety capability assessment system including three capability dimensions: safety perception, safety description, and safety reasoning. It can comprehensively measure the capabilities of the model from these three dimensions, objectively reflect the model's actual performance in the field of construction safety, and thus accurately identify the model's strengths and weaknesses in different scenarios and capabilities. It provides a scientific basis for selecting and optimizing multimodal large-scale models in the engineering construction field. It can not only help engineering construction units select the most suitable multimodal large-scale model, but also provide improvement directions for model developers, promoting the application and development of multimodal artificial intelligence technology in construction safety management. Secondly, by designing standardized multimodal construction safety assessment instructions, a standardized assessment process is achieved, reducing the influence of subjective factors in the assessment process and improving the reliability and repeatability of the assessment results. Thirdly, through a clear score calculation formula, the performance of different models can be intuitively compared, as can the performance gaps of the same model in different scenarios. This provides a quantitative basis for engineering construction units to select appropriate models, allowing for corresponding fine-tuning and optimization of model capabilities for relevant hidden danger scenarios. Finally, the multimodal large-scale model construction safety capability assessment method for the engineering field in this embodiment of the present invention is highly practical and scalable. The construction safety assessment dataset and multimodal construction safety assessment instructions can be continuously updated and expanded according to construction safety management needs, thereby adapting to emerging construction scenarios and safety standards.

[0048] In one embodiment, a device for evaluating the construction safety capability of a multi-modal large-scale model in an engineering field is provided. Figure 1 The engineering field multi-modal large model construction safety capability assessment method in the embodiment shown corresponds one to one. Figure 2 As shown, the multimodal large-scale model construction safety capability assessment device for the engineering field includes a capability assessment system module 10, an assessment data set construction module 20, an assessment instruction design module 30, a model assessment execution module 40, and an assessment score calculation module 50. The functional modules are described in detail as follows:

[0049] Capacity Assessment System Module 10, for implementation Figure 1 Step S10 in the multimodal large-scale model construction safety capability assessment method for the engineering field in the illustrated embodiment is used to construct a multi-dimensional construction safety capability assessment system, establish a capability assessment framework including three dimensions of safety perception, safety description, and safety reasoning, and design corresponding evaluation indicators for each capability dimension.

[0050] Evaluation dataset building module 20 for execution Figure 1Step S20 in the engineering field multimodal large-scale model construction safety capability assessment method in the illustrated embodiment is used to collect construction safety multi-scene image data and perform image annotation to construct a construction safety assessment dataset.

[0051] Evaluation instruction design module 30, for executing Figure 1 Step S30 in the engineering field multimodal large-scale model construction safety capability assessment method in the illustrated embodiment is used to design standardized multimodal construction safety assessment instructions, and different assessment instructions are designed corresponding to different capability dimensions and construction scenarios.

[0052] Model evaluation execution module 40 is used to execute Figure 1 Step S40 in the method for evaluating the construction safety capability of a multimodal large model in the engineering field in the illustrated embodiment is used to select a plurality of multimodal large models, execute standardized multimodal construction safety evaluation instructions, and perform model evaluation on the plurality of multimodal large models respectively according to a multidimensional construction safety capability evaluation system.

[0053] Evaluation score calculation module 50, for executing Figure 1 Step S50 in the method for evaluating construction safety capabilities of a multimodal large-scale model in the engineering field in the illustrated embodiment is used to calculate the performance score of each model in each capability dimension and construction scenario.

[0054] Regarding the specific limitations of the multimodal large-scale model construction safety capability assessment device in the engineering field, please refer to the limitations of the multimodal large-scale model construction safety capability assessment method in the engineering field above, which will not be repeated here. The various modules in the above-mentioned multimodal large-scale model construction safety capability assessment device in the engineering field can be implemented in whole or in part through software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0055] The multimodal large-scale model construction safety capability assessment device for the engineering field provided in the embodiment of the present invention constructs a construction safety assessment dataset covering a variety of construction scenarios such as foundation pit engineering, scaffolding engineering, and edge protection, and has wide applicability. It also constructs a multidimensional construction safety capability assessment system including three capability dimensions: safety perception, safety description, and safety reasoning. It can comprehensively measure the capabilities of the model from these three dimensions, objectively reflect the actual performance of the model in the field of construction safety, and thus accurately identify the advantages and disadvantages of the model in different scenarios and capabilities. It provides a scientific basis for selecting and optimizing multimodal large-scale models in the engineering construction field, which can not only help engineering construction units select the most suitable multimodal large-scale model, but also provide improvement directions for model developers, promoting the application and development of multimodal artificial intelligence technology in construction safety management. Secondly, by designing standardized multimodal construction safety assessment instructions, a standardized assessment process is achieved, which reduces the influence of subjective factors in the assessment process and improves the reliability and repeatability of the assessment results. Thirdly, through a clear score calculation formula, the performance of different models can be intuitively compared, as can the performance gap of the same model in different scenarios. This provides a quantitative basis for engineering construction units to select appropriate models, allowing for corresponding fine-tuning and optimization of model capabilities for relevant hidden danger scenarios. Finally, the multimodal large-scale model construction safety capability assessment device for engineering fields in the embodiments of the present invention is highly practical and scalable. The construction safety assessment dataset and multimodal construction safety assessment instructions can be continuously updated and expanded according to construction safety management needs, thereby adapting to emerging construction scenarios and safety standards.

[0056] In one embodiment, a computer device is provided, whose internal structure diagram can be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with other external electronic devices via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the multi-modal large-scale model construction safety capability assessment method in the engineering field in the embodiment shown in 1.

[0057] In one embodiment, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, Figure 1 The functions or steps of the method for assessing construction safety capability of a multimodal large-scale model in an engineering field in the illustrated embodiment.

[0058] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0059] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0060] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Those skilled in the art may make various equivalent changes and improvements based on the above embodiment. Any equivalent changes or modifications made within the scope of the claims shall fall within the scope of protection of the present invention.

Claims

1. A method for assessing construction safety capabilities of a multimodal large-scale model in an engineering field, characterized by: include: Construct a multi-dimensional construction safety capability assessment system, establish a capability assessment framework covering three dimensions: safety perception, safety description, and safety reasoning, and design corresponding assessment indicators for each capability dimension; Collect and annotate construction safety multi-scene image data to build a construction safety assessment dataset; Design standardized multimodal construction safety assessment instructions, with different assessment instructions corresponding to different capability dimensions and construction scenarios; Selecting multiple multimodal large models, executing standardized multimodal construction safety assessment instructions and performing model assessments on the multiple multimodal large models according to a multi-dimensional construction safety capability assessment system; Calculate the performance scores of each model in each capability dimension and construction scenario.

2. The method for evaluating construction safety capability of a multimodal large-scale model in an engineering field according to claim 1, wherein: The corresponding evaluation indicators are designed for each capability dimension, including: Regarding safety perception capabilities, the designed evaluation indicators include hidden danger object perception and hidden danger relationship identification; In terms of safety description capability, the designed evaluation indicators include hidden danger judgment and hidden danger description; Regarding safety reasoning capabilities, the designed evaluation indicators include risk analysis, accident consequences, standard index, standard requirements and corrective measures.

3. The method for evaluating construction safety capability of a multimodal large-scale model in an engineering field according to claim 2, wherein: The construction safety multi-scene image data includes multiple construction scene images including foundation pit engineering, scaffolding engineering, edge protection, unsafe behaviors of personnel and civilized construction.

4. The method for evaluating construction safety capability of a multimodal large-scale model in an engineering field according to claim 3, wherein: When image annotation is performed on the construction safety multi-scene image data, the image annotation content includes hidden danger judgment, hidden danger description, hidden danger object, hidden danger relationship, risk analysis, accident consequences, standard index, standard requirements and corrective measures.

5. The method for evaluating construction safety capability of a multimodal large-scale model in an engineering field according to claim 4, wherein: Different assessment instructions are designed for different capability dimensions and construction scenarios, including: Aiming at security perception capabilities, we designed hidden danger object recognition instructions and hidden danger relationship recognition instructions. When the hidden danger object recognition instruction is executed, it can identify the security hidden danger objects in the image. When the hidden danger relationship recognition instruction is executed, it can identify the relationship between hidden danger objects and the status attributes of the hidden danger objects themselves. Aiming at the safety description capability, we designed hidden danger judgment instructions and hidden danger description instructions. When the hidden danger judgment instruction is executed, it can determine whether there are safety hazards in the image. When the hidden danger description instruction is executed, it can describe the hidden dangers in the image in detail. In view of safety reasoning ability, risk analysis instructions, accident consequence prediction instructions, safety standard reference instructions, standard requirement description instructions and corrective measures generation instructions are designed. Among them, when the risk analysis instruction is executed, it can analyze in detail the safety risks that may be brought about by the hidden danger. When the accident consequence prediction instruction is executed, it can predict the accident consequences that may occur if the hidden danger is not dealt with in a timely manner. When the safety standard reference instruction is executed, it can reference relevant national standards or industry specifications as a basis. When the standard requirement description instruction is executed, it can list in detail the specific requirements for hidden danger objects in the safety standards. When the corrective measures generation instruction is executed, it can propose effective corrective measures for the hidden danger.

6. The method for evaluating construction safety capability of a multimodal large-scale model in an engineering field according to claim 5, wherein: Calculate the performance scores of each model in various capability dimensions and construction scenarios, including: Formula (1) is used to calculate the average Capability Score of each model in each capability dimension, where C is the sample set of one of the capabilities of security perception, security description and security reasoning, and Nc is the number of samples in set C. Formula (2) is used to calculate the average score of each model in each construction scenario, Scenario Score, where S is a sample set of one type of scenario, including foundation pit engineering, scaffolding engineering, edge protection, unsafe behavior of personnel, and civilized construction. Ns is the number of samples in set S. The overall accuracy Safety Score of each model is calculated using formula (3), where N is the total number of samples, s i represents the score of sample i, 7. The method for evaluating construction safety capability of a multimodal large-scale model in an engineering field according to any one of claims 1 to 6, characterized in that: The multiple multimodal large models include GPT-4o, Claude 3.5Sonnet, Gemini-1.5-Pro, Awaker2.5_VL, Qwen2-VL, InternVL2.5, LLaVA-OneVision, VITA-1.5, Mini-Gemini-34B-HD, MiniCPM-V 2.6, LLaVA1.5-7B, DeepSeek-VL2, LLaVA-NeXT-LLaMA3, and mPLUG DocOwl 1.

5.

8. A multi-modal large-scale model construction safety capability assessment device in the engineering field, characterized in that: include: The capability assessment system module is used to build a multi-dimensional construction safety capability assessment system, establish a capability assessment framework including three dimensions: safety perception, safety description, and safety reasoning, and design corresponding assessment indicators for each capability dimension; An evaluation dataset construction module is used to collect and annotate construction safety multi-scene image data to construct a construction safety evaluation dataset. The assessment instruction design module is used to design standardized multimodal construction safety assessment instructions, and to design different assessment instructions for different capability dimensions and construction scenarios; A model evaluation execution module is used to select multiple multimodal large models, execute standardized multimodal construction safety evaluation instructions, and perform model evaluation on the multiple multimodal large models according to the multi-dimensional construction safety capability evaluation system; The evaluation score calculation module is used to calculate the performance score of each model in various capability dimensions and construction scenarios.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method for evaluating the construction safety capability of a multimodal large model in an engineering field as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for evaluating the construction safety capability of a multimodal large model in an engineering field as described in any one of claims 1 to 7 are implemented.

Images