Construction safety monitoring method and device based on multi-modal artificial intelligence model, electronic equipment and program product

Abstract

This invention discloses a construction safety monitoring method, device, electronic equipment, and program product based on a multimodal artificial intelligence model. The method includes: acquiring video frames from the construction site; inputting the video frames and prompts into a multimodal large-scale model to obtain a scene description of the construction area; acquiring a set of anomaly judgment prompts for the monitoring task, including multiple steps in a predetermined order; performing restrictive judgments on multiple steps sequentially to obtain judgment results; using a large language model to judge the first step based on the scene description of the construction area; determining the visual input for subsequent steps based on the judgment results of previous steps and inputting it into the multimodal large-scale model to obtain the corresponding scene description; using the large language model to judge subsequent steps based on the corresponding scene description until the judgment results of each step are obtained; and outputting structured safety monitoring results based on each judgment result. This construction safety monitoring method improves the targeting and reliability of construction safety monitoring.

Description

Technical Field

[0001] This invention relates to the field of safety monitoring, and in particular to a construction safety monitoring method, device, electronic equipment, and program product based on a multimodal artificial intelligence model. Background Technology

[0002] Currently, safety monitoring at construction sites is of great significance for ensuring the safety of workers and equipment.

[0003] However, construction sites are often complex and involve diverse types of work, resulting in a variety of safety hazards that need to be addressed. Existing monitoring solutions still have room for improvement in terms of the relevance, reliability, and traceability of monitoring results.

[0004] Therefore, it is desirable to provide an improved safety monitoring solution to enhance the reliability of safety monitoring at construction sites.

[0005] The background description is provided for the purpose of understanding the relevant technologies in this field and is not intended as an admission of prior art. Summary of the Invention

[0006] In this regard, the present invention aims to provide a construction safety monitoring method, device, electronic device and program product based on a multimodal artificial intelligence model, which can improve the pertinence and reliability of construction safety monitoring and make the anomaly judgment process more traceable.

[0007] In a first aspect, embodiments of the present invention provide a construction safety monitoring method based on a multimodal artificial intelligence model, which may include:

[0008] Acquire video frames from the construction site;

[0009] The video frame and preset prompt words are input into the multimodal large model to obtain the scene description of the construction area output by the multimodal large model. The preset prompt words include monitoring task information and construction area information used to indicate the construction area in the construction site. The scene description is used to describe the construction scene located in the construction area in the video frame.

[0010] Obtain an anomaly determination prompt set corresponding to the monitoring task information, wherein the anomaly determination prompt set includes multiple verification steps arranged in a predetermined order, and in the multiple verification steps, the determination of a later verification step depends on the determination result of an earlier verification step.

[0011] According to the predetermined order, the multiple verification steps are subject to limited judgment to obtain the judgment result of each verification step; wherein, for the first verification step, the large language model judges the first verification step based on the scene description for the construction area; for subsequent verification steps, the visual input for the judgment of the subsequent verification step is determined based on the judgment result of the previous verification step, the visual input is input into the multimodal large model to obtain the scene description corresponding to the visual input output by the multimodal large model, and the large language model judges the subsequent verification step based on the scene description corresponding to the visual input, until the judgment result of each verification step is obtained, wherein the judgment range of each verification step is limited by the prompt information corresponding to the verification step in the abnormal judgment prompt set;

[0012] Based on the judgment results of each of the verification steps, a structured security monitoring result is output, wherein the structured security monitoring result includes a judgment field corresponding to each of the verification steps.

[0013] In some embodiments, the step of determining the visual input for the subsequent verification step based on the determination result of the preceding verification step, inputting the visual input into a multimodal large model to obtain a scene description corresponding to the visual input output by the multimodal large model, and having a large language model perform a determination in the subsequent verification step based on the scene description corresponding to the visual input, includes at least a portion of the following:

[0014] Based on the target location indicated by the determination result of the previous verification step, an image region corresponding to the target location is cropped from the image on which the previous verification step is based. When the previous verification step is the first verification step, the image is one or more of the video frames. When the previous verification step is not the first verification step, the image is the visual input of the previous verification step.

[0015] The image region is magnified to obtain a magnified image, and the magnified image is determined as the visual input for the judgment in the subsequent verification step;

[0016] The magnified image is input into the multimodal large model to obtain the local scene description of the target part output by the multimodal large model, which is used as the scene description corresponding to the visual input;

[0017] The local scene description is determined by the large language model based on the subsequent verification step.

[0018] In some embodiments, the video frames include video frames captured from multiple perspectives of the construction site.

[0019] The process of determining the visual input for the subsequent verification step based on the judgment result of the previous verification step, inputting the visual input into a multimodal large model to obtain the scene description corresponding to the visual input output by the multimodal large model, and having the large language model perform the judgment for the subsequent verification step based on the scene description corresponding to the visual input, includes at least a portion of the following:

[0020] Based on the determination result of the prior verification step, a second perspective video frame is determined from the video frames of the multiple perspectives, wherein the second perspective is different from the first perspective of the video frame corresponding to the prior verification step.

[0021] Based on one or more video frames captured from the second perspective, determine the visual input used for the determination in the subsequent verification step;

[0022] The visual input is input into the multimodal large model to obtain the scene description from the second perspective output by the multimodal large model, which is used as the scene description corresponding to the visual input.

[0023] The large language model determines the scene description from the second perspective based on the subsequent verification step.

[0024] In some embodiments, the step of determining the visual input for the subsequent verification step based on the determination result of the preceding verification step, inputting the visual input into a multimodal large model to obtain a scene description corresponding to the visual input output by the multimodal large model, and having a large language model perform a determination in the subsequent verification step based on the scene description corresponding to the visual input, includes at least a portion of the following:

[0025] Based on the determination result of the previous verification step, video frames collected in the second time period are determined from the video frames collected at the construction site at different time periods, wherein the second time period is different from the first time period of the video frames corresponding to the previous verification step;

[0026] Based on one or more video frames within the second time period, determine the visual input for the determination in the subsequent verification step;

[0027] The visual input is input into the multimodal large model to obtain the scene description of the second time period output by the multimodal large model, which is used as the scene description corresponding to the visual input.

[0028] The large language model determines the scene description in the second time period based on the subsequent verification step.

[0029] In some embodiments, for multiple consecutive subsequent verification steps, determining the visual input for the determination of the subsequent verification step based on the determination result of the preceding verification step includes:

[0030] Based on the determination result of the previous verification step, video frames collected in the second time period are determined from the video frames collected at the construction site at different time periods, wherein the second time period is later than the first time period of the video frames corresponding to the previous verification step;

[0031] Determining the visual input for the plurality of consecutive subsequent verification steps based on one or more video frames within the second time period specifically includes: based on the first target location indicated by the determination result of the first verification step in the plurality of consecutive verification steps, cropping a first image region corresponding to the first target location from one or more video frames within the second time period and enlarging the first image region to obtain a first enlarged image, which serves as the visual input for the determination of the second verification step; based on the second target location indicated by the determination result of a non-first verification step, cropping a second image region corresponding to the second target location from the first enlarged image and enlarging the second image region to obtain a second enlarged image, which serves as the visual input for the determination of the next verification step, until a preset condition is met.

[0032] In some embodiments, for multiple consecutive subsequent verification steps, determining the visual input for the determination of the subsequent verification step based on the determination result of the preceding verification step includes:

[0033] Based on the determination result of the previous verification step, video frames collected in the second time period are determined from the video frames collected at the construction site at different time periods, wherein the second time period is later than the first time period of the video frames corresponding to the previous verification step;

[0034] Determining the visual input for the plurality of consecutive subsequent verification steps based on one or more video frames within the second time period specifically includes: taking one or more video frames acquired from another perspective within the second time period as the visual input for the subsequent verification step based on the determination result of the previous verification step in the plurality of consecutive verification steps, until a preset condition is met.

[0035] In some embodiments, the video frames include video frames captured from multiple perspectives of the construction site, focusing on the same construction area.

[0036] The step of inputting the video frames and preset prompts into a multimodal large model to obtain a scene description of the construction area output by the multimodal large model includes:

[0037] The video frames captured from each of the aforementioned perspectives are input into the multimodal large model along with the preset prompt words to obtain multiple scene descriptions corresponding to the multiple perspectives;

[0038] For the first verification step, the large language model determines the first verification step based on the scenario description of the construction area, including:

[0039] The multiple scene descriptions are input into the large language model, and the large language model performs the restricted determination based on each scene description to obtain multiple preliminary determination results corresponding to the multiple scene descriptions.

[0040] Determine the consistency among the multiple scenario descriptions;

[0041] Based on the consistency among the multiple scenario descriptions, the preliminary judgment results are fused to obtain the judgment result of the first verification step.

[0042] In a second aspect, embodiments of the present invention provide a construction safety monitoring device, which may include:

[0043] The acquisition module is configured to acquire video frames from the construction site.

[0044] The scene description module is configured to input the video frame and preset prompt words into a multimodal large model to obtain a scene description of the construction area output by the multimodal large model. The preset prompt words include monitoring task information and construction area information used to indicate the construction area in the construction site. The scene description is used to describe the construction scene located in the construction area in the video frame.

[0045] The set acquisition module is configured to acquire an anomaly judgment prompt set corresponding to the monitoring task information. The anomaly judgment prompt set includes multiple verification steps arranged in a predetermined order. In the multiple verification steps, the judgment of a later verification step depends on the judgment result of an earlier verification step.

[0046] The restricted judgment module is configured to perform restricted judgments on the multiple verification steps in the predetermined order to obtain the judgment results of each verification step. Specifically, for the first verification step, the large language model judges the first verification step based on the scene description of the construction area. For subsequent verification steps, based on the judgment results of the previous verification steps, a visual input for judging the subsequent verification steps is determined. The visual input is input into the multimodal large model to obtain the scene description corresponding to the visual input output by the multimodal large model. The large language model then judges the subsequent verification steps based on the scene description corresponding to the visual input, until the judgment results of each verification step are obtained. The judgment range of each verification step is limited by the prompt information corresponding to that verification step in the abnormal judgment prompt set.

[0047] The result output module is configured to output structured security monitoring results based on the judgment results of each of the verification steps, wherein the structured security monitoring results include judgment fields corresponding to each of the verification steps.

[0048] In a third aspect, embodiments of the present invention provide an electronic device, characterized in that it includes: a processor and a memory storing a computer program, the processor being configured to implement the method as described in the first aspect when executing the computer program.

[0049] In a fourth aspect, embodiments of the present invention provide a program product including a computer program, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0050] The construction safety monitoring method of this invention obtains a scene description for the construction area by inputting video frames of the construction site along with preset prompts containing monitoring task information and construction area information into a multimodal large model. Then, an anomaly judgment prompt set including multiple verification steps arranged in a predetermined order and each subsequent verification step depending on the judgment result of the previous verification step is used by the large language model to progressively make restricted judgments on the scene description along the verification chain. The visual input on which the subsequent verification step is based is further determined by the judgment result of the previous verification step, and finally, a structured safety monitoring result is output. The method described above guides the model's focus by using construction area information, limits the sequential dependence between multiple verification steps, and determines the visual input step by step based on the judgment results. Through the cooperation and mutual support of these means, the general large model, which is originally easily distracted by irrelevant content and has divergent output, can carry out standardized verification around the construction area of ​​concern, along the established verification chain, and be traceable step by step. This overcomes the model's attention divergence and omission of details caused by the generalized processing of the entire picture in known solutions. At the same time, it avoids making judgments on subsequent verification items before the previous verification item is established, which would lead to contradictory conclusions. Therefore, it improves the pertinence, reliability, and effectiveness of safety monitoring in complex construction scenarios.

[0051] In a further embodiment of the invention, based on the target location indicated by the judgment result of the previous verification step, a corresponding image area is cropped from the image on which the previous verification step is based and magnified as visual input. This cropping and magnification are performed progressively in multiple rounds along multiple verification steps, so that the visual input is gradually focused on different parts of the construction area. Thus, the above solution, through the progressively magnified visual input, makes local details clearly presented, thereby further ensuring the effectiveness of the judgment in the subsequent verification step. Therefore, this solution is particularly suitable for monitoring local details in complex construction environments, such as the gap between the tongue and the hook body, which are difficult to identify within the entire field of view, improving the reliability of the determination of local and detailed verification points in the construction environment.

[0052] In a further embodiment of the present invention, visual input is determined in the perspective or time dimension based on the judgment result of the previous verification step. This allows for monitoring of the object when it is obscured or difficult to identify from the current perspective, or when the object to be verified is not within the current field of view. Furthermore, it allows for observation from subsequent video frames where the content to be verified has already been presented, even if the content to be verified has not yet appeared in the current video frame. Therefore, the above solution further overcomes the problems of obstruction under a single perspective and the difficulty of verifying sequentially occurring work steps within the same time period, further improving the reliability of the verification step's judgment.

[0053] Other optional features and technical effects of the embodiments of the present invention are partly described below and partly apparent from reading this document. Attached Figure Description

[0054] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The elements shown are not limited to the scale shown in the drawings, and the same or similar reference numerals in the drawings denote the same or similar elements, wherein:

[0055] Figure 1 A first exemplary flowchart of a construction safety monitoring method based on a multimodal artificial intelligence model according to an embodiment of the present invention is shown;

[0056] Figure 2 A second exemplary flowchart of a construction safety monitoring method based on a multimodal artificial intelligence model according to an embodiment of the present invention is shown;

[0057] Figure 3 A third exemplary flowchart of a construction safety monitoring method based on a multimodal artificial intelligence model according to an embodiment of the present invention is shown;

[0058] Figure 4A fourth exemplary flowchart of a construction safety monitoring method based on a multimodal artificial intelligence model according to an embodiment of the present invention is shown;

[0059] Figure 5 A fifth exemplary flowchart of a construction safety monitoring method based on a multimodal artificial intelligence model according to an embodiment of the present invention is shown;

[0060] Figure 6 A sixth exemplary flowchart of a construction safety monitoring method based on a multimodal artificial intelligence model according to an embodiment of the present invention is shown;

[0061] Figure 7 An exemplary block diagram of a construction safety monitoring device based on a multimodal artificial intelligence model according to an embodiment of the present invention is shown; and

[0062] Figure 8 An exemplary structural diagram of an electronic device capable of implementing the method according to an embodiment of the present invention is shown. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0064] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects.

[0065] In this embodiment of the invention, a multimodal large model refers to a large-scale machine learning model capable of simultaneously accepting and understanding multiple modal inputs such as images and text, and outputting the understanding results in the form of text, etc., such as, but not limited to, a Vision-Language Model (VLM). By way of explanation and not limitation, in this embodiment of the invention, both the multimodal large model and the large language model can be pre-trained models, which can be invoked through prompts without requiring retraining or fine-tuning for specific security monitoring tasks; the multimodal large model and the large language model can be implemented by the same or different models, or by the same or different model instances or model routes, and this embodiment of the invention does not impose any restrictions on this.

[0066] In this embodiment of the invention, in different verification steps of the same method, the multimodal large model used to generate scene descriptions and the large language model used for judgment can be routed to the same or different models or model instances, and this embodiment of the invention does not impose any restrictions on this. In this embodiment of the invention, the determination of visual input (including cropping, magnification, viewpoint switching, and time period switching, etc.) driven by the judgment result of the previous verification step is implemented as the processing flow of this embodiment of the invention itself, and does not depend on the characteristics of the multimodal large model or the large language model itself.

[0067] In this embodiment of the invention, the construction site includes various work sites that require safety monitoring. In a specific example, the construction site is the construction site of a highway reconstruction and expansion project. The construction content at the construction site may include, but is not limited to, hoisting operations, edge protection operations, roadbed construction, and pavement paving. This embodiment of the invention does not impose any limitations on these aspects.

[0068] As mentioned earlier, safety monitoring at construction sites often requires focusing on potential safety hazards within complex operational scenarios. Due to the complexity of construction site environments and the diversity of operational types, the safety hazards that need attention also vary in form, placing higher demands on safety monitoring. In an interpretative rather than restrictive sense, this invention recognizes the following challenges in safety monitoring at construction sites: First, the objects requiring key verification at construction sites are often relatively localized and detailed parts of the monitoring image, such as the anti-disengagement tongue on a crane hook, or even the gap between the tongue and the hook body. These details occupy a small proportion of the overall monitoring image and are therefore difficult to identify. Simultaneously, key verification items such as missing edge protection or hook detachment often have a sequential dependency in the actual verification process; only after confirming the existence of the structure to be verified can the details of that structure be further verified. Furthermore, construction sites may contain frames, columns, or temporarily parked construction equipment, and the objects to be verified may be obscured due to different viewing angles or may not yet be visible because the actions being verified occur at different times, thus placing high demands on the monitoring perspective and timing.

[0069] Based on an understanding of the difficulties encountered at construction sites, this invention recognizes that known monitoring solutions cannot meet the safety monitoring needs of construction sites. Specifically, some known solutions involve processing the monitoring images along with rather general instructions through a model. In this approach, the model is easily distracted by irrelevant content in the image, making it difficult to consistently focus attention on the specific areas of interest, and consequently, to make stable judgments on localized, detailed checkpoints.

[0070] Furthermore, some known solutions process the items to be verified in a parallel manner, ignoring the inherent dependencies between these items. This approach raises questions about the feasibility and efficiency of solutions that disregard practical engineering experience and requirements. Specifically, in such solutions, the model may make judgments about subsequent items before the previous one has been established, leading to contradictory conclusions. For example, judging whether there is a gap between the anti-detachment tongue and the hook body before confirming the existence of the anti-detachment tongue could easily result in contradictory judgments and monitoring failure. Moreover, this solution cannot guarantee a systematic and traceable verification process. Additionally, this approach uses a one-time analysis of the entire image, making it easy for existing models to overlook details, resulting in unreliable monitoring conclusions and failing to meet the requirements of practical engineering implementation.

[0071] To address this issue, this invention provides a construction safety monitoring scheme based on a multimodal artificial intelligence model, which effectively alleviates or overcomes the aforementioned problems. In this embodiment, by guiding the multimodal large model to focus on the construction area of ​​interest using construction area information at the input side, and constraining the judgment with multiple verification steps arranged in a predetermined order and interdependent, particularly by determining the visual input for subsequent verification steps based on the judgment results of earlier verification steps, details that are difficult to discern in the overall view can be clearly presented through progressively focused visual input. Furthermore, the judgment results of each verification step correspond to the judgment fields of the output structured safety monitoring results, greatly improving the feasibility and reliability of the artificial intelligence model for monitoring construction sites.

[0072] The construction safety monitoring method according to embodiments of the present invention can be used for construction safety monitoring in industrial construction areas such as, but not limited to, building construction areas, municipal engineering construction areas, and highway and bridge construction areas. It can also be used for monitoring the safety status of workers and machinery in scenarios such as construction sites, industrial parks, factories, and yards, as well as for other application scenarios requiring real-time perception and monitoring of safety events at construction sites, such as safety inspections, on-site verification, and post-event traceability during construction. The construction safety monitoring method according to embodiments of the present invention can be executed by, but is not limited to, construction safety monitoring devices deployed at construction sites, construction safety monitoring systems, analysis components of video surveillance platforms, and servers associated with video surveillance systems.

[0073] The specific implementation methods of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings and specific examples.

[0074] In some embodiments, reference Figure 1The construction safety monitoring method according to the present invention may include the following steps S110, S120, S130, S140 and S150.

[0075] S110: Acquire video frames from the construction site.

[0076] In some embodiments, video frames of the construction site can be acquired by one or more image acquisition devices deployed at the construction site. In this embodiment, the image acquisition devices are, for example, but not limited to, cameras, PTZ cameras, or other video acquisition equipment.

[0077] In some embodiments, the video frames may include video frames captured from multiple perspectives at the construction site, or video frames captured at different times at the construction site. For explanation, video frames from multiple perspectives and at multiple times can subsequently be used to determine the visual input for judgment in dimensions such as space, perspective, or time, which will be described in detail below.

[0078] In some embodiments, a video stream of the construction site can be acquired, and one or more video frames can be extracted from the video stream according to a predetermined time window as objects for subsequent processing, which falls within the protection scope of the embodiments of the present invention.

[0079] S120: Input the video frame and preset prompts into the multimodal large model to obtain the scene description of the construction area output by the multimodal large model.

[0080] In some embodiments, preset prompts may include monitoring task information and construction area information for indicating construction areas in the construction site.

[0081] In some embodiments, monitoring task information can be used to indicate the object or matter of interest for this monitoring. In a specific example, monitoring task information is used to indicate that the anti-disengagement status of a crane hook should be monitored.

[0082] In some embodiments, construction area information can be used to guide the focus of the multimodal large model to the construction area of ​​interest within the construction site. In a specific example, construction area information is used to guide the large model's focus to the structural location where the crane hook is located. By way of explanation and not limitation, by combining monitoring task information and construction area information as model input, the multimodal large model can focus on the specific construction area indicated when generating scene descriptions, reducing interference from content unrelated to the safety hazards of interest in the image, thereby avoiding interference from such content in the scene description and making the resulting scene description more closely aligned with the monitoring task.

[0083] In some embodiments, the scene description can be used to describe the construction scene located within a construction area in the video frame. In this embodiment, the construction area is, for example, but not limited to, a lifting and hoisting operation area, an edge protection area, and a high-altitude operation area. In this embodiment, the scope of the construction area can be defined by textual descriptions in preset prompts and / or by applying a region of interest to the video frame; this embodiment of the invention does not impose such limitations.

[0084] In some embodiments, the construction area can be defined in a multi-dimensional manner. In some embodiments, the construction area can be defined by a first definition and a second definition.

[0085] In some embodiments, the first limiting method may include a textual description of the construction area using construction area information in a preset prompt. In this embodiment, the first limiting method may also be referred to as a first natural language limiting method. In a specific example, the construction area is limited to a crane hook structure using text.

[0086] In some embodiments, the second limiting method may include applying a region of interest (ROI) to the video frame of the input multimodal large model, corresponding to the construction area. In this embodiment, the second limiting method may also be referred to as a second image region limiting. In a specific example, the video frame is cropped according to the ROI, or a mask is applied to the area outside the ROI.

[0087] As an explanation rather than a limitation, by combining semantic-level textual descriptions with image-level regions of interest to define the construction area, the multimodal large model's ability to understand different modal information is fully utilized, thereby better constraining the multimodal large model's focus to the construction area of ​​interest.

[0088] S130: Obtain the set of anomaly judgment prompts corresponding to the monitoring task information.

[0089] In some embodiments, the anomaly detection prompt set may include multiple verification steps arranged in a predetermined order. In this embodiment, the determination of a later verification step depends on the determination result of an earlier verification step.

[0090] In some embodiments, each verification step in the anomaly determination prompt set corresponds to an item to be verified, and the verification steps are arranged in a predetermined order to form a verification chain, and the determination of a later verification step is based on the determination result of an earlier verification step.

[0091] In one specific embodiment, an anomaly detection prompt set is used to check for deficiencies in the protection of the elevated bridge's edge work area. Multiple verification steps are performed in a predetermined order, including: a first verification step checking if a protected edge area exists in the screen; a second verification step checking if guardrails are installed in that edge area; and a third verification step checking if the guardrails are continuous without gaps or collapses. For explanation, only after confirming the existence of an edge area requiring protection is the verification of guardrail installation conducted; and only after confirming the installation of guardrails is the verification of their continuity without gaps conducted. Thus, there is a logical dependency between the verification steps, forming a sequential verification chain. In the above process, based on the edge area determined in the previous verification steps, the area is cropped and enlarged to check the continuity of the guardrails, and the verification of each section is sequentially advanced along the edge in a spatial or temporal dimension.

[0092] In another specific embodiment, an anomaly detection prompt set is used to check the anti-detachment mechanism of the crane hook. Multiple verification steps are performed in a predetermined order, including: a first verification step to check if the hook is visible; a second verification step to check if the anti-detachment tongue exists; and a third verification step to check if there is a gap between the tongue and the hook body. For explanation, the presence of the anti-detachment tongue is only checked after confirming that the hook is visible; and the presence of the anti-detachment tongue is only checked after confirming that there is a gap between the tongue and the hook body.

[0093] In some embodiments, the step of obtaining the set of anomaly judgment prompts corresponding to the monitoring task information may include: determining the construction scenario type corresponding to the monitoring task information, and determining the set of anomaly judgment prompts mapped to the construction scenario type as the set of anomaly judgment prompts corresponding to the monitoring task information based on the pre-established mapping relationship between the construction scenario type and the set of anomaly judgment prompts.

[0094] In some embodiments, the construction scenario type may include at least one of hoisting operation, edge protection operation and high-altitude operation. For explanation, different construction scenario types may be pre-associated with their own set of anomaly judgment prompts, so as to facilitate the expansion of the corresponding set of anomaly judgment prompts when adding a new construction task type.

[0095] In an optional embodiment, the anomaly detection prompt set can also be dynamically loaded according to the current construction plan stage. As an explanation and not a limitation, taking road construction as an example, the construction plan stage may include at least one of pile foundation construction, roadbed construction, pavement paving, and guardrail installation. The anomaly detection prompt sets corresponding to different construction plan stages may be different from each other, so as to check the matters appropriate to the stage at different construction plan stages.

[0096] In an optional embodiment, video frames can be input into a multimodal large model for work status recognition to obtain the current work status corresponding to the construction area. Based on a preset work state machine, a subset of verification steps corresponding to the current work status is activated from multiple verification steps. In this embodiment, the work state machine can define multiple work states and a subset of verification steps corresponding to each work state; the subsets of verification steps corresponding to different work states are different. In some embodiments, the restriction determination described below can be performed based on each verification step in the activated subset of verification steps.

[0097] In some embodiments, if a change in the current operation status is detected based on subsequently acquired video frames, the activated subset of verification steps can be updated according to the changed operation status. In a specific example, for hoisting operations, the operation state machine can define operation states for hook lifting, translation, and hook lowering, and their corresponding subsets of verification steps: in the hook lifting state, the focus is on verifying the hook and anti-disengagement tongue; in the translation state, the focus is on verifying whether personnel have passed under the hoisted object. In this embodiment, the subset of verification steps to be verified can be dynamically scheduled based on the perceived operation status, rather than executing all verification steps uniformly.

[0098] In an interpretive and not restrictive sense, this invention recognizes that construction site operations are often dynamic processes involving multiple stages. The key items requiring verification and the objects available for verification on the screen often differ at different stages. Therefore, if all verification steps in the anomaly detection prompt set are executed equally across all stages without distinction, firstly, for verification steps not applicable to the current stage, the corresponding verification objects may not yet be displayed on the screen, making reliable judgments impossible or even leading to conclusions that do not reflect reality. For example, checking whether personnel have passed under the hoisted object while it is still in the hook-raising stage and before it has been lifted and moved lacks a corresponding operational context for this verification. Secondly, executing all verification steps unrelated to the current stage indiscriminately incurs unnecessary processing costs and may interfere with the judgment of truly relevant safety hazards at the current stage with irrelevant verification results.

[0099] To address this, the method described in the above embodiments of the present invention identifies the current work status by inputting video frames into a multimodal large model, and dynamically activates a subset of verification steps corresponding to the current work status using a preset work state machine. Furthermore, it updates the activated subset of verification steps accordingly when a change in the work status is detected. This ensures that limited judgments always focus on the verification steps that truly need to be checked under the current work status, avoiding unreliable or meaningless judgments on verification steps that are not applicable to the current work stage, and reducing processing overhead and interference caused by irrelevant verifications. Moreover, as the work progresses, the focus of the verification can shift accordingly with the change in work status, ensuring that monitoring continuously aligns with the actual progress of the work, thereby further improving the targeting of monitoring and the reliability of monitoring results during dynamic operations.

[0100] S140: Perform constraint determination on multiple verification steps in a predetermined order to obtain the determination result of each verification step.

[0101] In some embodiments, the scope of judgment for each verification step is limited to the prompt information corresponding to that verification step in the anomaly judgment prompt set. In this embodiment, in other words, when judging a certain verification step, the judgment is limited to the scope of the items to be verified corresponding to that verification step, and no judgment is made on items outside that scope. For explanation, this allows the judgment of each verification step to focus on the predetermined verification items and facilitates the correspondence between the judgment results of each verification step and the judgment fields output subsequently.

[0102] In some embodiments, for the first verification step, the large language model can determine the first verification step based on the scene description of the construction area to obtain the determination result of the first verification step. In a specific embodiment, for the verification of the anti-disengagement of the crane hook in the hoisting operation, the large language model determines whether the hook is visible based on the scene description of the construction area and the prompt information corresponding to the first verification step, and obtains the location of the hook if the hook is determined to be visible.

[0103] In some embodiments, the video frames may include video frames captured from multiple perspectives of the construction site, each targeting the same construction area. This allows for collaborative judgment of the first verification step using information from multiple perspectives.

[0104] Accordingly, in some embodiments, reference is made to Figure 2 Step S120 may include the following step S210.

[0105] S210: Input the video frames collected from each perspective and the preset prompt words into the multimodal large model to obtain multiple scene descriptions corresponding to multiple perspectives.

[0106] In some embodiments, video frames captured from different viewpoints may contain the same object to be checked.

[0107] In some embodiments, such as after step S210 above, the step of determining the first verification step by the large language model based on the scene description for the construction area may include the following steps S220, S230 and S240.

[0108] S220: Input multiple scene descriptions into the large language model, and the large language model performs restricted judgments based on each scene description to obtain multiple preliminary judgment results corresponding to the multiple scene descriptions.

[0109] In some embodiments, for each scene description in multiple scene descriptions corresponding to multiple perspectives, the large language model can make a judgment on the first verification step based on the prompt information corresponding to the first verification step in the anomaly judgment prompt set, thereby obtaining the preliminary judgment result corresponding to the scene description.

[0110] S230: Determine the consistency among multiple scenario descriptions.

[0111] In some embodiments, a consistency comparison can be performed on descriptions of the same verification step in multiple scenario descriptions to determine whether the situations observed from different perspectives are consistent. In some embodiments, the consistency comparison may include: for the same verification step, comparing whether the descriptions of different perspectives regarding the existence of the verification object corresponding to that verification step are consistent, and / or comparing whether their descriptions regarding whether the verification object is abnormal are consistent. By way of explanation and not limitation, the consistency comparison here refers to the situation recorded in each scenario description itself, which is different from the preliminary determination made based on each scenario description in step S220 above.

[0112] S240: Based on the consistency among multiple scenario descriptions, the preliminary judgment results are fused to obtain the judgment result of the first verification step.

[0113] In some embodiments, if the verification objects corresponding to the verification step are consistent across multiple scenario descriptions, the same preliminary judgment result among the preliminary judgment results can be determined as the judgment result of the verification step. In some embodiments, if multiple scenario descriptions are consistent but the preliminary judgment results are not the same, the majority of the preliminary judgment results can be accepted, or the preliminary judgment result with higher step-level confidence can be accepted. The step-level confidence will be described in detail below.

[0114] In some embodiments, when the objects to be verified for the verification step are inconsistent across multiple scenario descriptions, the preliminary judgment result corresponding to the viewpoint where the object to be verified is not obstructed, the image is clearer, or the step-level confidence level is higher can be accepted as the judgment result of the verification step. In an optional embodiment, if the viewpoint to be accepted cannot be determined, the judgment result of the verification step can be marked as pending review.

[0115] In an interpretive and not restrictive sense, the method of the above embodiments of the present invention can correct or verify the preliminary judgment result of the verification step when the preliminary judgment result of a certain viewpoint is unreliable due to factors such as occlusion, reflection or viewing angle, by using the preliminary judgment result of other viewpoints, which helps to reduce misjudgment and missed detection caused by occlusion under a single viewpoint.

[0116] In some embodiments, for the subsequent verification step, the visual input for the determination of the subsequent verification step can be determined based on the determination result of the preceding verification step, and the visual input can be input into the multimodal large model to obtain the scene description corresponding to the visual input output by the multimodal large model.

[0117] In some embodiments, the visual input includes image content determined based on the determination result of a prior verification step. By way of explanation and not limitation, unlike methods that process the same image content all at once, the method of this embodiment dynamically determines the visual input used in subsequent verification steps based on the determination result of a prior verification step, such that the determination of subsequent verification steps is based on specifically determined visual input.

[0118] In some embodiments, a large language model may make a determination on the subsequent verification steps based on the scene description corresponding to the visual input, until the determination results of each verification step are obtained.

[0119] In some embodiments, unlike the scene description of the construction area generated based on the acquired video frames, the scene description corresponding to the visual input is generated based on the visual input determined by the judgment result of the prior verification step.

[0120] In some embodiments, based on the determination result of the preceding verification step, corresponding image content can be selected or processed from video frames collected at the construction site as visual input for the subsequent verification step. In some embodiments, the selection or processing can be performed on at least one of the spatial, temporal, and perspective dimensions.

[0121] In some embodiments, spatially, the field of view and magnification of the video frame can be adjusted, for example, by cropping and magnifying a corresponding image region from the video frame based on the determination result of the previous verification step. In some embodiments, temporally, the acquisition time period of the video frame can be adjusted, for example, by acquiring video frames from the construction site at a different time period. In some embodiments, visually, the acquisition viewing angle of the video frame can be adjusted, for example, by acquiring video frames from the construction site from a different viewing angle. It is understood that the above dimensions can be used individually or in combination, as will be described in detail below in conjunction with further embodiments.

[0122] In some embodiments, visual input for the determination in the post-verification step can be determined in a spatial dimension. Accordingly, in some embodiments, reference is made to... Figure 3 Step S140 may include at least part of steps S310, S320, S330, and S340 described below. As an explanation, and not a limitation, during the multiple verification steps, the visual input upon which the determination in the later verification steps is based is determined through the following steps.

[0123] S310: Based on the target region indicated by the determination result of the previous verification step, crop out the image region corresponding to the target region from the image on which the previous verification step is based.

[0124] In some embodiments, the determination result of the pre-verification step may also indicate the target region being verified in that verification step. In this embodiment, the determination result may include the location information of the target region in the image on which the pre-verification step is based. In some embodiments, the location information may include, but is not limited to, region coordinates, bounding boxes, or relative position description information.

[0125] In some embodiments, the cropping region containing the target portion can be determined based on location information. In one example, the cropping region is determined based on the bounding box of the target portion, and then the corresponding image content is extracted from the image on which the previous verification step was based, according to the cropping region. By way of explanation and not limitation, the region coordinates of the cropping region used for cropping may also be recorded for subsequent traceability of the determination process.

[0126] In some embodiments, when the pre-verification step is the first verification step, the image it is based on is one or more video frames; when the pre-verification step is not the first verification step, the image it is based on is the visual input of the pre-verification step.

[0127] S320: Enlarge the image region to obtain an enlarged image, and determine the enlarged image as the visual input for judgment in the subsequent verification step.

[0128] In some embodiments, the image region may be directly magnified or optionally upsampled or resampled, etc., and the embodiments of the present invention do not limit this.

[0129] S330: Input the magnified image into the multimodal large model to obtain the local scene description of the target area output by the multimodal large model, which is used as the scene description corresponding to the visual input.

[0130] As an explanation and not a limitation, a local scene description can describe the details of the target area in a more granular way than a scene description generated based on the image before magnification. The local scene description is the scene description corresponding to the aforementioned visual input.

[0131] S340: The local scene description is determined by the large language model based on the post-verification step.

[0132] In some embodiments, the large language model can determine the description of a local scene based on the prompt information in the anomaly judgment prompt set that corresponds to the subsequent verification step.

[0133] In one specific embodiment, a verification process is performed to check the anti-disengagement mechanism of the hoisting hook during hoisting operations. The first verification step determines the visibility of the hook based on the scene description of the construction area and identifies the location of the hook. Then, the image area where the hook is located is cropped from the video frame and enlarged. The enlarged local image is input into a multimodal large model to obtain a local scene description, which is used by the second verification step to determine whether the anti-disengagement tongue exists. Based on the location of the tongue indicated by the determination result of the second verification step, the image area of ​​the tongue root is cropped from the corresponding image and enlarged, which is used by the third verification step to determine whether there is a gap between the tongue and the hook body.

[0134] In some embodiments, during the determination process of the plurality of verification steps performed in a predetermined order, the cropping and magnification processing may be determined based on the determination results of each verification step and executed in multiple progressive rounds according to the plurality of verification steps, so that the visual input of each verification step progressively focuses on a given part in the construction area.

[0135] As an explanation, and not a limitation, unlike methods that process the entire image at once, the step-by-step focusing process of the above embodiments of the present invention can locate and crop the image based on the judgment results of the preceding verification steps. Furthermore, the cropping and magnification of the embodiments of the present invention are driven by the judgment results of each verification step and are executed iteratively in multiple rounds along the verification chain formed by multiple verification steps, allowing the visual input to be focused step-by-step on different parts of the construction area and clearly presenting the structure of the monitored object. Moreover, the above solution is particularly suitable for identifying details that are difficult to discern within the framing of the entire video frame, such as the gap between the tongue and the hook body, thereby supporting the judgment in subsequent verification steps.

[0136] In an optional embodiment, determining the visual input in the spatial dimension may further include adjusting the image resolution for the construction area based on the determination result of the previous verification step, and using the image content at that resolution as the visual input, which falls within the protection scope of the embodiments of the present invention.

[0137] As mentioned earlier, video frames can include video frames captured from multiple perspectives of the construction site, each targeting the same construction area. This allows for collaborative judgment in subsequent verification steps by utilizing information from multiple perspectives.

[0138] Accordingly, in some embodiments, reference is made to Figure 4 Step S140 may include at least part of the following steps S410, S420, S430, and S440. As an explanation, and not a limitation, during the multiple verification steps, the visual input upon which the determination in the later verification steps is based is determined through the following steps.

[0139] S410: Based on the determination result of the previous verification step, determine the video frame of the second perspective from the video frames of multiple perspectives.

[0140] In some embodiments, the second perspective is different from the first perspective of the video frame corresponding to the pre-verification step. In some embodiments, the video frames of the second perspective and the first perspective may contain the same object to be verified. In other words, the second perspective and the first perspective are video frames observing the same object to be verified from different perspectives.

[0141] In one specific embodiment, when verifying hook-related matters from a first perspective, since the hook is partially obscured by the boom or slings from the first perspective, the view can be switched to another perspective that presents the hook to verify the anti-disengagement tongue on the same hook. For explanation, switching to a second perspective helps to observe the same object to be verified from another angle when it is obscured or difficult to identify from the first perspective, and provides a usable basis for subsequent judgment.

[0142] S420: Determine the visual input for the decision in the subsequent verification step based on one or more video frames captured from the second perspective.

[0143] In some embodiments, the difference between step S420 and step S320 is that the visual input used for the subsequent verification step in step S420 is taken from the video frame captured from the second perspective, rather than from the local image obtained by cropping and enlarging the video frame from the perspective corresponding to the previous verification step.

[0144] In some embodiments, when determining the visual input, one or more video frames captured from the second perspective can be directly used as the visual input. Alternatively, the video frames from the second perspective can be further cropped and enlarged in the spatial dimension to combine the perspective dimension with the spatial dimension, thereby focusing on the target part under the second perspective while switching to the second perspective. This falls within the protection scope of the embodiments of the present invention.

[0145] S430: Input the visual input into the multimodal large model to obtain the scene description in the second perspective output by the multimodal large model, which is used as the scene description corresponding to the visual input.

[0146] S440: The large language model determines the scene description from the second perspective based on the post-verification step.

[0147] In some embodiments, steps S430 and S440 can refer to the corresponding descriptions of steps S330 and S340 above, respectively. The difference is that in step S430, the visual input of the input multimodal large model is taken from the second perspective, and the obtained scene description is from the second perspective; in step S440, the large language model is based on the scene description from the second perspective.

[0148] In other embodiments, the video frames from the second and first perspectives may contain different objects to be verified. In this embodiment, in other words, the object to be verified in the subsequent verification step is different from the object verified in the preceding verification step. As an explanation, and not a limitation, the object to be verified in the subsequent verification step may not be within the framing of the first perspective, or the first perspective may have difficulty presenting the object, making verification impossible from the first perspective. In this case, based on the determination result of the preceding verification step, the system can switch to a second perspective that can present the other object, thereby verifying that other object. In one specific embodiment, after confirming that the crane is in a lifting operation state from the first perspective, the system switches to a second perspective that can present the workers to verify whether the workers in the work area are wearing the prescribed safety protective equipment.

[0149] In optional embodiments, the perspective switching can also be performed at the granular level. In some embodiments, when the judgment result of a target verification step in a plurality of verification steps meets a preset perspective switching condition, and / or when the step-level confidence level corresponding to the verification step is lower than a preset switching threshold, the visual input used for the judgment of the target verification step can be switched to the image content corresponding to the construction area in a video frame acquired from another perspective, and the judgment of the target verification step can be re-executed based on the switched visual input. In this embodiment, the judgment results of the remaining verification steps in the plurality of verification steps remain unchanged. By way of explanation and not limitation, this allows for supplementary observation from another perspective only at verification steps where it is truly necessary, without affecting the conclusions of the remaining steps in the entire verification chain.

[0150] As previously mentioned, in some embodiments, visual input for the decision in the subsequent verification step can be determined in the time dimension. For explanation, this allows for collaborative decision-making in the subsequent verification step using information from multiple points in time / time periods.

[0151] Accordingly, in some embodiments, reference is made to Figure 5 Step S140 may include at least part of the following steps S510, S520, S530, and S540. As an explanation, and not a limitation, during the multiple verification steps, the visual input upon which the determination in the later verification steps is based is determined through the following steps.

[0152] S510: Based on the judgment results of the previous verification steps, determine the video frames collected in the second time period from the video frames collected at the construction site at different time periods.

[0153] In some embodiments, the second time period differs from the first time period of the video frames corresponding to the preceding verification step. In some embodiments, the second time period may be later than the first time period.

[0154] S520: Determine the visual input for the decision in the subsequent verification step based on one or more video frames within the second time period.

[0155] S530: Input the visual input into the multimodal large model to obtain the scene description of the second time period output by the multimodal large model, which is used as the scene description corresponding to the visual input.

[0156] S540: The large language model determines the scene description in the second time period based on the post-verification step.

[0157] In some embodiments, the specific descriptions of steps S520 to S540 are similar to those of steps S420 to S440 described above. The difference is that the video frames used in step S520 are taken from the second time period rather than the second perspective. Accordingly, the scene description obtained in step S530 is the scene description under the second time period, and the large language model in step S540 is based on the scene description under the second time period.

[0158] As explained rather than limited, as previously stated, construction site operations often consist of multiple sequential steps or actions. Therefore, the content to be checked in a later verification step may not appear until later than the time period corresponding to the earlier verification step. Consequently, the video frames corresponding to the time period of the earlier verification step may not contain image content that the later verification step can use for judgment. In response, the above embodiments of the present invention determine a second time period to be checked in the later verification step based on the judgment result of the earlier verification step, and take the video frames of this second time period as the visual input for the later verification step. This allows the later verification step to make judgments based on image content that has indeed appeared, thereby progressively covering each sequentially occurring work step along the time dimension.

[0159] It is understood that the steps related to the spatial dimension, temporal dimension, and perspective dimension involved in the above embodiments of the present invention can be implemented individually or in combination between different verification steps. As an explanation and not a limitation, the steps can be implemented by progressively cropping and enlarging only in the spatial dimension, such as focusing progressively on the hook, the tongue, and the base of the tongue for a hook; or by advancing only along the temporal dimension, taking video frames from subsequent time periods to verify sequentially occurring actions; or by using a spatiotemporal combination method, coordinating different verification steps, for example, first advancing along the temporal dimension to take subsequent time periods, and then continuously cropping and enlarging within those time periods, or first cropping and enlarging, and then advancing along the temporal dimension to switch to another part, which falls within the protection scope of the embodiments of the present invention.

[0160] In some embodiments, reference Figure 6 For multiple consecutive subsequent verification steps, the step of determining the visual input for the determination of the subsequent verification step based on the determination result of the previous verification step may include the following steps S610 and S620.

[0161] S610: Based on the judgment results of the previous verification steps, determine the video frames collected in the second time period from the video frames collected at the construction site at different time periods.

[0162] In some embodiments, the second time period is later than the first time period of the video frame corresponding to the preceding verification step. For explanation, if the determination result of the preceding verification step confirms that the previous operation has been completed or has occurred, the time period corresponding to the subsequent operation that is later than the first time period can be determined as the second time period.

[0163] S620: Determines one or more video frames within the second time period as visual input for multiple consecutive post-verification steps.

[0164] In some embodiments, multiple consecutive post-verification steps may share the same second time period. In this embodiment, in other words, after a progression in the time dimension, the visual input of the multiple consecutive post-verification steps can be determined step-by-step within the second time period without having to re-determine each verification step in the time dimension separately.

[0165] In some embodiments, step S620 may include the following steps A1 and A2.

[0166] A1: Based on the first target location indicated by the determination result of the first verification step in a series of consecutive verification steps, a first image region corresponding to the first target location is cropped from one or more video frames in the second time period, and the first image region is magnified to obtain a first magnified image, which serves as the visual input for the determination of the second verification step.

[0167] A2: Based on the second target location indicated by the judgment result of a non-first verification step, a second image region corresponding to the second target location is cropped from the first magnified image, and the second image region is magnified to obtain a second magnified image, which serves as the visual input for the judgment of the next verification step, and so on until the preset conditions are met.

[0168] In one specific embodiment, a determination is made regarding the connection and fastening stage of guardrail installation in highway reconstruction and expansion projects. After confirming that the guardrail panels have been installed in place based on the determination results of the previous verification steps, the time period corresponding to the connection and fastening stage, which is later than the guardrail panel installation stage, is determined as the second time period. Then, the first image area corresponding to the connection part is cropped from the video frame of the second time period and enlarged to check whether the bolts of the connection part are complete. Then, the second image area corresponding to a single bolt is further cropped from the enlarged image and enlarged to check whether the bolt is tightened in place.

[0169] In an interpretative rather than restrictive sense, this invention recognizes that in on-site verification, certain consecutive verification items occur or are presented only in subsequent time periods later than the time periods corresponding to previous verification steps, and further fall into increasingly localized areas or require observation from different perspectives. In this regard, if processing is only performed in the temporal dimension without further processing in the spatial or visual dimensions, although video frames of subsequent time periods where the content has already been presented may be obtained, relatively localized details may still be difficult to discern or may still be obscured. Conversely, if processing is only performed in the spatial or visual dimensions without progressing in the temporal dimension, since the content to be verified has not yet occurred or been presented in the video frames corresponding to the time periods of previous verification steps, further cropping, magnification, or changing perspectives is impossible. In other words, for such consecutive verification items that occur in subsequent time periods, fall into increasingly localized areas, or require observation from different perspectives, processing in a single dimension is often insufficient to provide sufficient visual input for judgment. In this regard, in the above embodiments of the present invention, the visual input for the subsequent verification step is obtained by further cropping and enlarging the magnified image obtained in the previous verification step, rather than being re-cropped from the original video frames of the second time period. This allows multiple consecutive visual inputs in subsequent verification steps to be nested and focused on progressively more localized areas within the second time period. Therefore, the method of the above embodiments of the present invention can obtain verifiable visual inputs one by one along the verification chain, advancing synergistically in both the temporal and spatial dimensions, further ensuring the effectiveness and feasibility of the monitoring.

[0170] In some embodiments, for multiple consecutive subsequent verification steps, the step of determining the visual input for the subsequent verification step based on the determination result of the preceding verification step may further include: determining video frames collected in a second time period from video frames collected at different time periods at the construction site based on the determination result of the preceding verification step, wherein the second time period is later than the first time period of the video frames corresponding to the preceding verification step; and determining one or more video frames in the second time period as the visual input for multiple consecutive subsequent verification steps.

[0171] In some embodiments, the step of determining one or more video frames within a second time period as visual input for a plurality of consecutive subsequent verification steps may include: taking one or more video frames acquired from another perspective within the second time period as visual input for the determination of a subsequent verification step based on the determination result of the previous verification step in a plurality of consecutive verification steps, and so on until a preset condition is met.

[0172] In this embodiment, the difference between the above steps and the aforementioned steps A1 and A2 is that, after proceeding to the second time period, the visual input for the multiple consecutive subsequent verification steps is determined by gradually changing the viewpoint along the visual dimension within the second time period, rather than by progressively cropping and enlarging along the spatial dimension. As an explanation, and not a limitation, the above approach is particularly suitable for situations where the consecutive verification items in subsequent time periods need to be observed separately from different perspectives.

[0173] In some embodiments, the preset conditions may include at least one of the following: the number of progressive rounds has reached a preset number, the number of verification steps processed has reached a preset number, and all verification steps have been completed. In some embodiments, the preset conditions may also include at least one of the following: the multimodal large model or large language model has identified a preset target object or target feature, the magnification in the spatial dimension has reached a preset upper limit, and the image clarity or resolution of the visual input has met the judgment requirements.

[0174] S150: Output structured security monitoring results based on the judgment results of each verification step.

[0175] In some embodiments, the structured security monitoring results may include decision fields corresponding to each verification step.

[0176] In some embodiments, the judgment fields corresponding to each verification step may include at least: a region field indicating the construction area, an anomaly indication field indicating whether the judgment result of the verification step indicates the existence of an anomaly, a confidence field indicating the confidence level of the judgment result of the verification step, and a situation description field describing the anomaly corresponding to the verification step. For explanation, each verification step may correspond to a judgment field in the structured safety monitoring result, so that the monitoring result not only provides a conclusion on whether an anomaly exists, but also provides traceable judgment information corresponding to each verification step.

[0177] In some embodiments, the determination result of each verification step may also include the step-level confidence level corresponding to the verification step, and the step-level confidence level may be carried in the aforementioned confidence level field corresponding to the verification step.

[0178] In some embodiments, the step-level confidence of a preceding verification step can be propagated to a subsequent verification step in a predetermined order, and the step-level confidence of a subsequent verification step can be adjusted based on the step-level confidence of the preceding verification step. If the step-level confidence of a preceding verification step does not meet a preset confidence condition, the decision result of the subsequent verification step is marked as a conditional conclusion. As an explanation, and not a limitation, since the decision of a subsequent verification step is based on the decision of a preceding verification step, when the decision of the preceding verification step, which serves as a premise, is uncertain, the conclusion of the subsequent verification step, upon which it is based, should be retained accordingly.

[0179] In some embodiments, a composite confidence level can also be determined based on the step-level confidence levels of each verification step, for example, by taking the minimum value among the step-level confidence levels or the product of the step-level confidence levels. In this embodiment, the composite confidence level can be used to characterize the overall confidence level of the judgment results of multiple verification steps; if the composite confidence level is lower than a preset composite confidence level threshold, the structured security monitoring result can be marked as a result to be reviewed, and the composite confidence level can be incorporated into the structured security monitoring result.

[0180] In some embodiments, evidence on which the determination is based may be retained for each verification step. In some embodiments, for each verification step, the visual input used for the determination of the verification step and the region coordinates of the visual input in the video frame may be recorded as an evidence snapshot corresponding to the verification step; and the verification step, the determination result of the verification step, and the evidence snapshot corresponding to the verification step may be bound together to obtain a step-determination-evidence group corresponding to the verification step; and an evidence chain arranged in a predetermined order may be generated based on the step-determination-evidence group corresponding to each verification step.

[0181] In an optional embodiment, if the step-level confidence of a target verification step in a plurality of verification steps is lower than a preset confidence threshold, the manual annotation process corresponding to the target verification step can be triggered to obtain the manual annotation result corresponding to the target verification step.

[0182] In some embodiments, the prompt information corresponding to the target verification step in the anomaly detection prompt set can be optimized based on the manual annotation results. The optimized prompt information is then used to update the anomaly detection prompt set, which is used to perform restriction determination on subsequently acquired video frames. By way of explanation and not limitation, this allows for the closed-loop improvement of the prompt information corresponding to the verification step through manual annotation, enabling the anomaly detection prompt set to be gradually improved during use without requiring retraining of the entire model.

[0183] The following detailed description of the construction safety monitoring method based on a multimodal artificial intelligence model according to embodiments of the present invention is provided with reference to specific examples. However, it should be understood that the following examples are for illustrative purposes only, and the embodiments of the present invention are not limited thereto.

[0184] In one specific embodiment, a check is performed on personnel surrounding road machinery operations during highway reconstruction and expansion projects. As an explanation, during road machinery operations, it is necessary to check whether there are personnel in the blind spots around the machinery to avoid mechanical injury. However, from a perspective along the road direction, the blind spots at the rear or sides of the machinery are often obscured by the machinery itself, making it difficult to observe whether there are personnel in these blind spots. Therefore, in this embodiment, the anomaly detection prompt set includes, in a predetermined order, a check step to verify whether the machinery is in operation, and a check step to verify whether there are personnel in the blind spots around the machinery.

[0185] In this embodiment, after the pre-verification step confirms that the machinery is in operation based on the scene description from the corresponding viewpoint, the post-verification step, based on the determination result, uses video frames captured from another viewpoint that can present the blind spot as visual input. A multimodal large model generates a scene description from this viewpoint, and a large language model then verifies whether there are personnel in the blind spot. Thus, by using an appropriate viewpoint in the perspective dimension, the problem of the blind spot around the machinery being obscured by the machinery itself in the original viewpoint is overcome, allowing the previously difficult-to-observe blind spot to be verified from an appropriate location.

[0186] In another specific embodiment, a hook anti-derailment check is performed for (crane) lifting operations in highway reconstruction and expansion projects. For explanation, the crane hook is located at the end of the boom and occupies a small proportion of the overall monitoring screen. The anti-derailment tongue, a component located at the hook opening to prevent the sling from slipping off, is smaller than the hook itself, and the gap between the tongue and the hook body is even more minute. Such objects are difficult to identify within the overall field of view. In this embodiment, the anomaly detection prompt set includes, in a predetermined order, a check step to verify whether the crane hook is visible, a check step to verify the presence of the hook's anti-derailment tongue, and a check step to verify whether there is a gap between the anti-derailment tongue and the hook body. The first verification step confirms the visibility of the hook based on the scene description of the construction area and determines its location in the image. The image area containing the hook is then cropped from the video frame and magnified as the visual input for the next verification step. A large language model further verifies the presence of the anti-detachment tongue on the hook. After confirming the presence of the anti-detachment tongue, the image area containing the tongue is cropped from the magnified image and further magnified as the visual input for the next verification step. The large language model then uses this to verify whether there is a gap between the tongue and the hook body. Thus, by progressively focusing from the hook and tongue to the gap along the verification chain, the system clearly reveals previously difficult-to-discern minute gaps, thereby determining whether the hook poses a risk of detachment.

[0187] In another specific embodiment, the inspection focuses on the guardrail installation work in highway reconstruction and expansion projects. For explanation, guardrail installation is typically carried out sequentially in stages such as post driving, guardrail panel installation, and connection tightening. The content to be inspected in each stage differs: the post driving stage focuses on verifying whether the posts are vertical and driven in correctly; the guardrail panel installation stage focuses on verifying whether the guardrail panels are installed in place and whether the overlaps are correct; and the connection tightening stage focuses on verifying whether the bolts at the connection points are complete and tightened. In this embodiment, the anomaly detection prompt set includes inspection steps arranged in a predetermined order, corresponding to the post driving, guardrail panel installation, and connection tightening stages respectively. After confirming the corresponding content of the previous stage, the subsequent inspection step, based on the detection result, uses video frames from the time period corresponding to the next stage as visual input along the time dimension, and the large language model uses this as the basis to inspect the corresponding content of the next stage. Thus, by advancing sequentially along the verification chain in the time dimension and combining it with the gradual expansion in the spatial dimension when necessary, the verification content occurring at different stages is verified at the appropriate time.

[0188] Therefore, the construction safety monitoring method provided in this embodiment of the invention can have at least some of the following technical effects.

[0189] The construction safety monitoring method of this invention obtains a scene description for the construction area by inputting video frames of the construction site along with preset prompts containing monitoring task information and construction area information into a multimodal large model. Then, an anomaly judgment prompt set including multiple verification steps arranged in a predetermined order and each subsequent verification step depending on the judgment result of the previous verification step is used by the large language model to progressively make restricted judgments on the scene description along the verification chain. The visual input on which the subsequent verification step is based is further determined by the judgment result of the previous verification step, and finally, a structured safety monitoring result is output. The method described above guides the model's focus by using construction area information, limits the sequential dependence between multiple verification steps, and determines the visual input step by step based on the judgment results. Through the cooperation and mutual support of these means, the general large model, which is originally easily distracted by irrelevant content and has divergent output, can carry out standardized verification around the construction area of ​​concern, along the established verification chain, and be traceable step by step. This overcomes the model's attention divergence and omission of details caused by the generalized processing of the entire picture in known solutions. At the same time, it avoids making judgments on subsequent verification items before the previous verification item is established, which would lead to contradictory conclusions. Therefore, it improves the pertinence, reliability, and effectiveness of safety monitoring in complex construction scenarios.

[0190] In a further embodiment of the invention, based on the target location indicated by the judgment result of the previous verification step, a corresponding image area is cropped from the image on which the previous verification step is based and magnified as visual input. This cropping and magnification are performed progressively in multiple rounds along multiple verification steps, so that the visual input is gradually focused on different parts of the construction area. Thus, the above solution, through the progressively magnified visual input, makes local details clearly presented, thereby further ensuring the effectiveness of the judgment in the subsequent verification step. Therefore, this solution is particularly suitable for monitoring local details in complex construction environments, such as the gap between the tongue and the hook body, which are difficult to identify within the entire field of view, improving the reliability of the determination of local and detailed verification points in the construction environment.

[0191] In a further embodiment of the present invention, visual input is determined in the perspective or time dimension based on the judgment result of the previous verification step. This allows for monitoring of the object when it is obscured or difficult to identify from the current perspective, or when the object to be verified is not within the current field of view. Furthermore, it allows for observation from subsequent video frames where the content to be verified has already been presented, even if the content to be verified has not yet appeared in the current video frame. Therefore, the above solution further overcomes the problems of obstruction under a single perspective and the difficulty of verifying sequentially occurring work steps within the same time period, further improving the reliability of the verification step's judgment.

[0192] In a further embodiment of the invention, for multiple consecutive subsequent verification steps, the process first advances to a subsequent time period in the time dimension, and then, within that time period, it progressively crops and enlarges the visual input along the spatial dimension, or gradually changes the viewing angle along the visual perspective dimension. Therefore, the above solution can effectively monitor verification items that occur in subsequent time periods, fall into progressively more localized areas, or require observation from different perspectives, thereby further improving its applicability to complex verification scenarios.

[0193] In a further embodiment of the present invention, multiple scene descriptions are obtained by inputting video frames collected from multiple perspectives of the same construction area with preset prompts into a multimodal large model, and the preliminary judgment results are fused based on the consistency among the multiple scene descriptions. Therefore, the above solution can correct or verify the preliminary judgment results from other perspectives when the preliminary judgment result from a certain perspective is unreliable due to factors such as occlusion, reflection, or viewing angle. This further reduces misjudgments and missed detections caused by single-view occlusion, thereby enhancing the robustness of the judgment process and the reliability of the judgment results.

[0194] In a further embodiment of the present invention, the step-level confidence of the preceding verification step is propagated along the verification chain to the subsequent verification step and a composite confidence is determined accordingly. This allows the dependency between the preceding and subsequent steps in the chain verification to be reflected in the confidence level. Consequently, questionable judgment results can be identified based on the confidence level. Furthermore, by binding the visual input and its regional coordinates used in each verification step with the judgment result, an evidence chain is generated. This enables the judgment process of the entire verification chain to be traced back and audited step by step, further improving the reliability and traceability of the monitoring results.

[0195] In an embodiment of the present invention, reference is made to Figure 7 Furthermore, a construction safety monitoring device 700 based on a multimodal artificial intelligence model is also provided. The construction safety monitoring device 700 may include an acquisition module 710, a scene description module 720, a set acquisition module 730, a restricted judgment module 740, and a result output module 750.

[0196] In some embodiments, the acquisition module 710 is configured to acquire video frames from the construction site.

[0197] In some embodiments, the scene description module 720 is configured to input the video frame and preset prompt words into a multimodal large model to obtain a scene description for the construction area output by the multimodal large model. The preset prompt words include monitoring task information and construction area information for indicating the construction area in the construction site. The scene description is used to describe the construction scene located in the construction area in the video frame.

[0198] In some embodiments, the set acquisition module 730 is configured to acquire an anomaly judgment prompt set corresponding to the monitoring task information, wherein the anomaly judgment prompt set includes multiple verification steps arranged in a predetermined order, wherein the judgment of a later verification step depends on the judgment result of an earlier verification step.

[0199] In some embodiments, the restricted determination module 740 is configured to perform restricted determination on the plurality of verification steps in the predetermined order to obtain the determination result of each verification step; wherein, for the first verification step, the large language model determines the first verification step based on the scene description for the construction area; for subsequent verification steps, the visual input for the determination of the subsequent verification steps is determined according to the determination result of the previous verification steps, the visual input is input into the multimodal large model to obtain the scene description corresponding to the visual input output by the multimodal large model, and the large language model determines the subsequent verification steps based on the scene description corresponding to the visual input, until the determination result of each verification step is obtained, wherein the determination range of each verification step is limited by the prompt information corresponding to the verification step in the abnormal determination prompt set.

[0200] In some embodiments, the result output module 750 is configured to output a structured security monitoring result based on the determination result of each of the verification steps, wherein the structured security monitoring result includes a determination field corresponding to each of the verification steps.

[0201] In this embodiment of the invention, the specific implementation of the construction safety monitoring device 700, and the details of each unit performing the corresponding steps in the aforementioned method and its optional implementations, can be found in the detailed description of the aforementioned construction safety monitoring method embodiment, and will not be repeated here.

[0202] Accordingly, the construction safety monitoring device and its components, modules, units, and features described in the above embodiments can be incorporated into the construction safety monitoring method of the present invention in a non-contradictory manner, and will not be elaborated here. Conversely, the construction safety monitoring method and its steps, sub-steps, and features described in the embodiments of the present invention can also be incorporated into the construction safety monitoring device of the present invention in a non-contradictory manner.

[0203] It is understood that the division of the units in the construction safety monitoring device 700 is a logical functional division. In actual implementation, there may be other division methods. For example, multiple units may be combined or integrated into one module, and each unit may be implemented by software, hardware, or a combination of software and hardware. All of the above fall within the scope of this invention.

[0204] In an embodiment of the present invention, an electronic device is also provided, which includes a processor and a memory storing a computer program, the processor being configured to implement the method of any embodiment of the present invention when running the computer program.

[0205] Figure 8 A schematic diagram is shown of a method or electronic device 800 that can be used to implement embodiments of the present invention. In some embodiments, the number of electronic devices may be more or less than the number shown. In some embodiments, it can be implemented using a single or multiple electronic devices. It can also be implemented using cloud or distributed electronic devices in some embodiments.

[0206] like Figure 8As shown, the electronic device 800 includes a processor 801 and a memory 802. The processor executes programs stored in the memory, which, when executed by a computer, can implement the methods, steps, or functions described in the above embodiments. The processor 801 may include various types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. The processor 801 and the memory 802 are interconnected via a bus 803. Input / output (I / O) interfaces may also be connected to the bus 803. The systems, devices, modules, or units illustrated in the above embodiments can be implemented by a computer or its associated components.

[0207] Although not shown, in this embodiment of the invention, a program product is provided, including a computer program that, when executed by a processor, implements the method of any embodiment of the invention.

[0208] Although not shown, in an embodiment of the invention, a storage medium is provided storing a computer program configured to be run to implement the method of any embodiment of the invention.

[0209] Storage media in embodiments of the present invention include articles that are permanent and non-permanent, removable and non-removable, capable of storing information by any method or technology. Examples of storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0210] The methods, programs, systems, apparatuses, etc., in embodiments of the present invention can be executed or implemented in one or more networked computers, or practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks can be performed by remote processing devices connected via a communication network.

[0211] Those skilled in the art will understand that the embodiments described in this specification can be provided as methods, systems, or computer program products. Therefore, those skilled in the art will realize that the functional modules / units or controllers and related method steps described in the above embodiments can be implemented in software, hardware, or a combination of both.

[0212] Unless explicitly stated otherwise, the actions or steps of the methods and procedures described in the embodiments of the present invention do not necessarily have to be performed in a specific order and can still achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0213] This document describes several embodiments of the present invention; however, for the sake of brevity, the descriptions of the embodiments are not exhaustive, and identical or similar features or parts between the embodiments may be omitted. In this document, "one embodiment," "some embodiments," "example," "specific example," or "some examples" refers to embodiments applicable to at least one, but not all, of the present invention. The above terms do not necessarily refer to the same embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of the different embodiments or examples.

[0214] The exemplary systems and methods of the present invention have been specifically shown and described with reference to the above embodiments, which are merely examples of the best mode for implementing the systems and methods. Those skilled in the art will understand that various changes can be made to the embodiments of the systems and methods described herein without departing from the spirit and scope of the invention as defined in the appended claims when implementing the systems and / or methods.

Claims

1. A construction safety monitoring method based on a multimodal artificial intelligence model, characterized in that, The method includes: Acquire video frames from the construction site; The video frame and preset prompt words are input into the multimodal large model to obtain the scene description of the construction area output by the multimodal large model. The preset prompt words include monitoring task information and construction area information used to indicate the construction area in the construction site. The scene description is used to describe the construction scene located in the construction area in the video frame. Obtain an anomaly determination prompt set corresponding to the monitoring task information, wherein the anomaly determination prompt set includes multiple verification steps arranged in a predetermined order, and in the multiple verification steps, the determination of a later verification step depends on the determination result of an earlier verification step. According to the predetermined order, the multiple verification steps are subject to limited judgment to obtain the judgment result of each verification step; wherein, for the first verification step, the large language model judges the first verification step based on the scene description for the construction area; for subsequent verification steps, the visual input for the judgment of the subsequent verification step is determined based on the judgment result of the previous verification step, the visual input is input into the multimodal large model to obtain the scene description corresponding to the visual input output by the multimodal large model, and the large language model judges the subsequent verification step based on the scene description corresponding to the visual input, until the judgment result of each verification step is obtained, wherein the judgment range of each verification step is limited by the prompt information corresponding to the verification step in the abnormal judgment prompt set; Based on the judgment results of each of the verification steps, a structured security monitoring result is output, wherein the structured security monitoring result includes a judgment field corresponding to each of the verification steps.

2. The construction safety monitoring method according to claim 1, characterized in that, The step of determining the visual input for the subsequent verification step based on the judgment result of the previous verification step, inputting the visual input into a multimodal large model to obtain the scene description corresponding to the visual input output by the multimodal large model, and having the large language model perform the judgment for the subsequent verification step based on the scene description corresponding to the visual input, includes at least a portion of the following: Based on the target location indicated by the determination result of the previous verification step, an image region corresponding to the target location is cropped from the image on which the previous verification step is based. When the previous verification step is the first verification step, the image is one or more of the video frames. When the previous verification step is not the first verification step, the image is the visual input of the previous verification step. The image region is magnified to obtain a magnified image, and the magnified image is determined as the visual input for the judgment in the subsequent verification step; The magnified image is input into the multimodal large model to obtain the local scene description of the target part output by the multimodal large model, which is used as the scene description corresponding to the visual input; The local scene description is determined by the large language model based on the subsequent verification step.

3. The construction safety monitoring method according to claim 1, characterized in that, The video frames include video frames captured from multiple perspectives of the construction site. The process of determining the visual input for the subsequent verification step based on the judgment result of the previous verification step, inputting the visual input into a multimodal large model to obtain the scene description corresponding to the visual input output by the multimodal large model, and having the large language model perform the judgment for the subsequent verification step based on the scene description corresponding to the visual input, includes at least a portion of the following: Based on the determination result of the prior verification step, a second perspective video frame is determined from the video frames of the multiple perspectives, wherein the second perspective is different from the first perspective of the video frame corresponding to the prior verification step. Based on one or more video frames captured from the second perspective, determine the visual input used for the determination in the subsequent verification step; The visual input is input into the multimodal large model to obtain the scene description from the second perspective output by the multimodal large model, which is used as the scene description corresponding to the visual input. The large language model determines the scene description from the second perspective based on the subsequent verification step.

4. The construction safety monitoring method according to claim 1, characterized in that, The step of determining the visual input for the subsequent verification step based on the judgment result of the previous verification step, inputting the visual input into a multimodal large model to obtain the scene description corresponding to the visual input output by the multimodal large model, and having the large language model perform the judgment for the subsequent verification step based on the scene description corresponding to the visual input, includes at least a portion of the following: Based on the determination result of the previous verification step, video frames collected in the second time period are determined from the video frames collected at the construction site at different time periods, wherein the second time period is different from the first time period of the video frames corresponding to the previous verification step; Based on one or more video frames within the second time period, determine the visual input for the determination in the subsequent verification step; The visual input is input into the multimodal large model to obtain the scene description of the second time period output by the multimodal large model, which is used as the scene description corresponding to the visual input. The large language model determines the scene description in the second time period based on the subsequent verification step.

5. The construction safety monitoring method according to any one of claims 1 to 4, characterized in that, For multiple consecutive subsequent verification steps, the step of determining the visual input for the subsequent verification step based on the determination result of the previous verification step includes: Based on the determination result of the previous verification step, video frames collected in the second time period are determined from the video frames collected at the construction site at different time periods, wherein the second time period is later than the first time period of the video frames corresponding to the previous verification step; Determining the visual input for the plurality of consecutive subsequent verification steps based on one or more video frames within the second time period specifically includes: based on the first target location indicated by the determination result of the first verification step in the plurality of consecutive verification steps, cropping a first image region corresponding to the first target location from one or more video frames within the second time period and enlarging the first image region to obtain a first enlarged image, which serves as the visual input for the determination of the second verification step; based on the second target location indicated by the determination result of a non-first verification step, cropping a second image region corresponding to the second target location from the first enlarged image and enlarging the second image region to obtain a second enlarged image, which serves as the visual input for the determination of the next verification step, until a preset condition is met.

6. The construction safety monitoring method according to any one of claims 1 to 4, characterized in that, For multiple consecutive subsequent verification steps, the step of determining the visual input for the subsequent verification step based on the determination result of the previous verification step includes: Based on the determination result of the previous verification step, video frames collected in the second time period are determined from the video frames collected at the construction site at different time periods, wherein the second time period is later than the first time period of the video frames corresponding to the previous verification step; Determining the visual input for the plurality of consecutive subsequent verification steps based on one or more video frames within the second time period specifically includes: taking one or more video frames acquired from another perspective within the second time period as the visual input for the subsequent verification step based on the determination result of the previous verification step in the plurality of consecutive verification steps, until a preset condition is met.

7. The construction safety monitoring method according to any one of claims 1 to 4, characterized in that, The video frames include video frames captured from multiple perspectives of the construction site, focusing on the same construction area. The step of inputting the video frames and preset prompts into a multimodal large model to obtain a scene description of the construction area output by the multimodal large model includes: The video frames captured from each of the aforementioned perspectives are input into the multimodal large model along with the preset prompt words to obtain multiple scene descriptions corresponding to the multiple perspectives; For the first verification step, the large language model determines the first verification step based on the scenario description of the construction area, including: The multiple scene descriptions are input into the large language model, and the large language model performs the restricted determination based on each scene description to obtain multiple preliminary determination results corresponding to the multiple scene descriptions. Determine the consistency among the multiple scenario descriptions; Based on the consistency among the multiple scenario descriptions, the preliminary judgment results are fused to obtain the judgment result of the first verification step.

8. A construction safety monitoring device, characterized in that, The device includes: The acquisition module is configured to acquire video frames from the construction site. The scene description module is configured to input the video frame and preset prompt words into a multimodal large model to obtain a scene description of the construction area output by the multimodal large model. The preset prompt words include monitoring task information and construction area information used to indicate the construction area in the construction site. The scene description is used to describe the construction scene located in the construction area in the video frame. The set acquisition module is configured to acquire an anomaly judgment prompt set corresponding to the monitoring task information. The anomaly judgment prompt set includes multiple verification steps arranged in a predetermined order. In the multiple verification steps, the judgment of a later verification step depends on the judgment result of an earlier verification step. The restricted judgment module is configured to perform restricted judgments on the multiple verification steps in the predetermined order to obtain the judgment results of each verification step. Specifically, for the first verification step, the large language model judges the first verification step based on the scene description of the construction area. For subsequent verification steps, based on the judgment results of the previous verification steps, a visual input for judging the subsequent verification steps is determined. The visual input is input into the multimodal large model to obtain the scene description corresponding to the visual input output by the multimodal large model. The large language model then judges the subsequent verification steps based on the scene description corresponding to the visual input, until the judgment results of each verification step are obtained. The judgment range of each verification step is limited by the prompt information corresponding to that verification step in the abnormal judgment prompt set. The result output module is configured to output structured security monitoring results based on the judgment results of each of the verification steps, wherein the structured security monitoring results include judgment fields corresponding to each of the verification steps.

9. An electronic device, characterized in that, It includes a processor and a memory storing a computer program, the processor being configured to implement the construction safety monitoring method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer program product, characterized in that, It includes a computer program, which, when executed by a processor, implements the construction safety monitoring method as described in any one of claims 1 to 7.

Images