Multi-modal chain reasoning data automatic construction method and device for power safety supervision scene and computer equipment

Abstract

The application relates to a multi-modal chain reasoning data automatic construction method, device and computer equipment for a power safety monitoring scene. The method comprises the following steps: target detection is performed on a safety monitoring image in a power operation scene to obtain a detection result of the safety monitoring image; label information corresponding to the safety monitoring image is acquired; the safety monitoring image and the detection result are fused to obtain a target label image; the target label image and instruction text are input into a preset reasoning model, and the preset reasoning model generates reasoning process information and a violation judgment result of the target label image according to a logical reasoning step indicated by the instruction text; and the safety monitoring image, the instruction text, the reasoning process information and the violation judgment result are combined into multi-modal sample data to construct a training data set. The method can improve the accuracy of violation behavior detection in the power operation scene.

Description

Technical Field

[0001] This application relates to the field of power safety monitoring technology, and in particular to a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for the automated construction of multimodal chain-based reasoning data for power safety monitoring scenarios. Background Technology

[0002] In the process of safety supervision during power construction, the automatic identification of violations at the work site is a crucial technical means to ensure safe production. Power construction operations are characterized by complex environments and numerous risk sources. Violations such as people standing under crane booms during hoisting operations and climbing ladders without assistance are typical high-risk violations, often leading to serious consequences in the event of an accident. Traditionally, the identification of such violations relies primarily on manual video inspections, which suffers from low efficiency, high missed detection rates, and difficulty in real-time response. Therefore, there is an urgent need to utilize intelligent technologies to achieve automatic and accurate identification of violations.

[0003] Currently, Convolutional Neural Networks (CNNs), such as the YOLO series or Faster R-CNN, are commonly used to detect targets such as personnel, crane booms, and ladders in construction site images. However, these methods can only output the category and location of discrete targets and cannot determine complex safety relationships such as "whether there are personnel staying under the crane boom during hoisting operations" or "whether there are personnel using ladders during uphill and high-altitude operations." This makes it difficult to accurately detect complex violations, resulting in low accuracy in detecting violations in power operation scenarios. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, device, computer equipment, computer-readable storage medium, and computer program product for the automated construction of multimodal chain-based reasoning data for power safety supervision scenarios, which can improve the accuracy of detecting violations in power operation scenarios.

[0005] Firstly, this application provides an automated method for constructing multimodal chain-based inference data for power safety monitoring scenarios, including:

[0006] Target detection is performed on safety monitoring images in power operation scenarios to obtain detection results for the safety monitoring images; the detection results are used to label target objects in the safety monitoring images.

[0007] Obtain the annotation information corresponding to the safety monitoring image; the annotation information includes semantic information used to indicate the type of work scenario, the status of the target object, and the conditions for triggering violations;

[0008] The security monitoring image and the detection result are fused to obtain a target labeled image, and the labeled information is input into a preset instruction template to obtain instruction text;

[0009] The target labeled image and the instruction text are input into a preset inference model. The preset inference model generates inference process information and violation judgment result of the target labeled image according to the logical inference steps indicated by the instruction text. The inference process information is used to describe the inference process of behavior judgment of the target labeled image.

[0010] The safety monitoring images, the instruction text, the reasoning process information, and the violation determination results are combined into multimodal sample data to construct a training dataset. The training dataset is used to fine-tune the multimodal large model to be trained, resulting in a trained multimodal large model. The trained multimodal large model is used to identify violations in power operation scenarios.

[0011] In one embodiment, the logical reasoning steps indicated by the instruction text include the following steps:

[0012] Identify the working environment and job type of the target labeled image;

[0013] Locate the target object corresponding to the task type in the work environment; analyze the positional relationship and interaction state between the target objects to obtain the scene description information of the target annotation image;

[0014] The scenario description information is matched with the power safety regulations in the preset rule base to obtain the matching result;

[0015] Based on the matching result, the violation determination result is generated, as well as the reasoning process information containing the logical reasoning path is generated.

[0016] In one embodiment, analyzing the positional relationships and interaction states between the target objects to obtain scene description information of the target labeled image includes:

[0017] Extract the two-dimensional visual features of each target object in the target labeled image;

[0018] Based on the two-dimensional visual features, the relative distance, relative height, and mutual occlusion relationships of each target object in three-dimensional space are analyzed to obtain scene description information of the target labeled image.

[0019] In one embodiment, before combining the security monitoring image, the instruction text, the reasoning process information, and the violation determination result into multimodal sample data, the method further includes:

[0020] The consistency verification is performed on the reasoning process information and the violation determination result to obtain the consistency verification result;

[0021] If the consistency verification result indicates that the verification has passed, the step of combining the security monitoring image, the instruction text, the reasoning process information, and the violation determination result into multimodal sample data is performed.

[0022] If the consistency check result indicates that the check fails, adjust the instruction template, input the annotation information into the adjusted instruction template, and obtain the updated instruction text;

[0023] The preset inference model regenerates the inference process information and the violation judgment result based on the target labeled image and the updated instruction text, until the consistency check is passed.

[0024] In one embodiment, the consistency verification of the reasoning process information and the violation determination result to obtain a consistency verification result includes:

[0025] The violation determination result is compared with the preset manually labeled answer using a preset verification model to obtain a first verification result; the first verification result is used to indicate whether the violation determination result is consistent with the manually labeled answer.

[0026] The reasoning process information is compared with the violation determination result using the preset reasoning model to obtain a second verification result; the second verification result is used to indicate whether the reasoning process information is consistent with the violation determination result.

[0027] If both the first verification result and the second verification result indicate consistency, the consistency verification result is determined to be a pass verification.

[0028] If at least one of the verification results in the first verification result and the second verification result is inconsistent, the consistency verification result is determined to be a failure.

[0029] In one embodiment, after combining the security monitoring image, the instruction text, the reasoning process information, and the violation determination result into multimodal sample data to construct a training dataset, the method further includes:

[0030] The multimodal large model to be trained is fine-tuned using the training dataset to obtain the trained multimodal large model.

[0031] The image of the power operation scene to be detected and the text of the behavior detection instruction are input into the trained multimodal large model to obtain the violation judgment result for the power operation scene image, as well as the inference process information corresponding to the violation judgment result.

[0032] Secondly, this application also provides an automated construction device for multimodal chain-based inference data for power safety monitoring scenarios, including:

[0033] The detection module is used to perform target detection on safety monitoring images in power operation scenarios to obtain the detection results of the safety monitoring images; the detection results are used to mark the target objects in the safety monitoring images.

[0034] The acquisition module is used to acquire the annotation information corresponding to the safety monitoring image; the annotation information includes semantic information indicating the type of operation scenario, the status of the target object, and the conditions for triggering violations;

[0035] The processing module is used to fuse the security monitoring image with the detection result to obtain a target labeled image, and to input the labeled information into a preset instruction template to obtain instruction text;

[0036] The judgment module is used to input the target labeled image and the instruction text into a preset inference model, and generate inference process information and violation judgment result of the target labeled image through the preset inference model according to the logical inference steps indicated by the instruction text; the inference process information is used to describe the inference process of judging the behavior of the target labeled image;

[0037] The construction module is used to combine the safety monitoring image, the instruction text, the reasoning process information, and the violation judgment result into multimodal sample data to construct a training dataset; the training dataset is used to fine-tune the multimodal large model to be trained to obtain a trained multimodal large model; the trained multimodal large model is used to identify violations in power operation scenarios.

[0038] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0039] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0040] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0041] The aforementioned method, device, computer equipment, computer-readable storage medium, and computer program product for automated construction of multimodal chain-based inference data for power safety monitoring scenarios, through target detection of safety monitoring images in power operation scenarios and fusion of the detection results with the original images to obtain target-annotated images, enhances the visual saliency of the target object's location, and provides a structured target location prior for subsequent inference models; it acquires annotation information containing three types of semantic anchors: operation scenario type, target object status, and violation triggering conditions, and transforms them into instruction text with prior condition constraints and inference path constraints using a preset instruction template, reducing annotation costs and inconsistencies compared to the traditional method of manually writing complete inference processes line by line; the target-annotated images and instruction text are input into a preset inference model. The model generates reasoning process information and violation judgment results containing complete reasoning paths according to the logical reasoning steps indicated by the instruction text. It explicitly decomposes the originally implicit visual judgment process into interpretable chain reasoning steps. Finally, it combines safety monitoring images, instruction text, reasoning process information and violation judgment results into multimodal sample data and constructs a training dataset. Each sample in the training dataset contains explicit chain reasoning information. This enables the multimodal large model obtained by fine-tuning based on the training dataset to improve from recognition based on appearance features to understanding and judgment based on reasoning process. In the power safety supervision scenario, it achieves robust understanding and interpretable judgment of multi-target spatial relationships, complex interactive behaviors and safety procedure logic, and improves the accuracy of violation detection in power operation scenarios. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is an application environment diagram of an automated construction method for multimodal chain-based inference data for power safety monitoring scenarios, as shown in one embodiment.

[0044] Figure 2 This is a flowchart illustrating an automated data construction method for multimodal chained reasoning in a power safety monitoring scenario, as shown in one embodiment.

[0045] Figure 3 This is a flowchart of a dataset construction method in one embodiment;

[0046] Figure 4This is a schematic diagram of the input and output of a multimodal large model during the inference phase in one embodiment;

[0047] Figure 5 This is a structural block diagram of a multimodal chain-based inference data automated construction device for power safety monitoring scenarios, as shown in one embodiment.

[0048] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] The multimodal chain-based inference data automation construction method for power safety monitoring scenarios provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.

[0051] Terminal 102 performs target detection on safety monitoring images in power operation scenarios to obtain detection results for the safety monitoring images; the detection results are used to label target objects in the safety monitoring images; terminal 102 acquires the labeling information corresponding to the safety monitoring images; the labeling information includes semantic information indicating the type of operation scenario, the status of the target object, and the conditions for triggering violations; terminal 102 fuses the safety monitoring images and detection results to obtain a target-labeled image, and inputs the labeling information into a preset instruction template to obtain instruction text; terminal 102 inputs the target-labeled image and instruction text into a preset inference model, and generates inference process information and violation judgment results for the target-labeled image according to the logical inference steps indicated by the instruction text through the preset inference model; the inference process information is used to describe the inference process of judging behavior in the target-labeled image; terminal 102 combines the safety monitoring images, instruction text, inference process information, and violation judgment results into multimodal sample data to construct a training dataset; the training dataset is used to fine-tune the multimodal large model to be trained to obtain a trained multimodal large model; the trained multimodal large model is used to identify violations in power operation scenarios.

[0052] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0053] In one exemplary embodiment, such as Figure 2 As shown, an automated method for constructing multimodal chain-based inference data for power safety monitoring scenarios is provided, and this method is applied to... Figure 1 Taking terminal 102 as an example, the explanation includes:

[0054] Step S202: Target detection is performed on the safety monitoring image in the power operation scenario to obtain the detection result of the safety monitoring image.

[0055] Among them, safety monitoring images can refer to image data collected by video monitoring equipment at power operation sites. Specifically, safety monitoring images originate from video streams collected by fixed cameras or mobile inspection equipment deployed at power construction sites, and are obtained by a terminal extracting keyframes from the video stream at a fixed frame rate.

[0056] The operational scenarios covered by the safety monitoring images include, but are not limited to, typical power safety monitoring scenarios such as hoisting operation areas, road construction areas, building construction areas, and industrial power pole operation areas. To ensure the diversity of subsequent model training data, the acquisition process of safety monitoring images takes into account various geographical environments such as cities, mountains, and plains, as well as various lighting and meteorological conditions such as sunny days, cloudy days, low light at night, and fog and haze.

[0057] The target objects refer to the key safety monitoring elements that need to be detected and monitored in power operation scenarios, including but not limited to workers, crane booms, ladders, safety helmets, safety belts, insulated tools, and work platforms. The specific category of the target objects is determined by the terminal from a preset category list based on the current operation scenario. For example, in a hoisting operation scenario, the target objects include the crane boom, the object being hoisted, and the workers; in an elevated work scenario, the target objects include the ladder, the workers climbing the ladder, and the people using the ladder.

[0058] For example, the terminal inputs a security monitoring image into a pre-trained object detection model, which then performs feature extraction and bounding box regression on the security monitoring image, outputting the position coordinates and category label of each target object in the image. The pre-trained object detection model can be a YOLO series model or a Faster R-CNN model. The specific network structure of the object detection model is selected according to the complexity and real-time requirements of the operation scenario, and this embodiment does not limit it.

[0059] The detection results are used to label target objects in security monitoring images. Optionally, the detection results are represented as a set of bounding boxes, each bounding box consisting of its top-left corner coordinates, width, height, and corresponding category label and confidence score in the image.

[0060] Optionally, before inputting the security monitoring image into the pre-trained target detection model, the terminal performs preprocessing operations on the security monitoring image. The preprocessing operations include two sub-steps: keyframe extraction and quality filtering. In the keyframe extraction sub-step, the terminal extracts candidate frames from the video stream at a preset frame rate (e.g., 1 frame per second or 2 frames per second) to avoid redundancy between adjacent frames. In the quality filtering sub-step, the terminal calculates the sharpness score of each candidate frame and the similarity score between adjacent candidate frames, removing blurry frames with sharpness scores below a preset sharpness threshold, and removing duplicate frames with similarity scores above a preset similarity threshold. This preprocessing operation can significantly reduce the computational load of subsequent processing without sacrificing scene information coverage.

[0061] Step S204: Obtain the annotation information corresponding to the security monitoring image.

[0062] The annotation information includes semantic information used to indicate the type of work scenario, the status of the target object, and the conditions for triggering violations.

[0063] Among them, the operation scenario type refers to the classification label of the power operation scenario corresponding to the safety monitoring image, such as hoisting operation, high-altitude operation, live-line operation, operation near live conductor, etc.

[0064] Among them, the target object status refers to the specific status of the target object in the safety monitoring image, such as the position status of the workers (located on the ground, on the platform, or in the middle of the ladder), the equipment status (the extension direction of the boom, the tilt angle of the ladder), and the equipment status (whether they are wearing safety helmets or safety belts).

[0065] Among them, the violation triggering condition refers to the logical condition that is pre-set for the current work scenario type and is used to determine whether there is a violation. For example, in the hoisting operation scenario, the violation triggering condition is "there are people staying under the boom", and in the climbing operation scenario, the violation triggering condition is "the height of the people on the ladder exceeds the preset height threshold and there are no people holding the ladder below".

[0066] In practice, the terminal acquires annotation information using a semantic anchor-based lightweight annotation method. Unlike traditional methods that require writing complete descriptions or reasoning processes for each sample, the semantic anchor-based lightweight annotation method only requires annotators to provide three types of key semantic information: scene category recognition results, core object existence judgment results, and risk trigger condition judgment results. The terminal receives these three types of semantic information input by annotators through the annotation tool, organizes them into structured annotation information, and stores it in the sample database. This method reduces the annotation workload from writing complete reasoning processes to simply selecting a few preset options, reducing the annotation time for a single image from minutes to seconds.

[0067] Step S206: The security monitoring image and the detection result are fused to obtain the target labeled image, and the labeled information is input into the preset instruction template to obtain the instruction text.

[0068] The fusion process involves the terminal visually synthesizing the security monitoring image with the bounding box information from the detection results. The bounding boxes are then overlaid on the original security monitoring image with preset colors and line widths, and corresponding category labels are added near the bounding boxes. The fused image is the target-labeled image. While retaining all pixel information from the original security monitoring image, the target-labeled image enhances the visual saliency of the target object's location, enabling the subsequent inference model to prioritize core targets such as workers, crane booms, and ladders during the perception phase.

[0069] The preset instruction template refers to a predefined text template used to transform annotation information into structured input for the inference model. The instruction template comprises four components: task description field, prior condition field, inference path constraint field, and output format constraint field. The task description field declares the objective of the current inference task; the prior condition field contains the task scenario type, target object status, and violation triggering conditions from the annotation information; the inference path constraint field specifies the analysis steps the inference model should follow during inference; and the output format constraint field specifies the format and fields of the inference model's output.

[0070] In practice, the process of the terminal inputting the annotation information into the preset instruction template to obtain the instruction text includes: extracting three types of semantic information from the annotation information: job scenario type, target object status, and violation triggering conditions; filling the extracted semantic information into the corresponding prior condition field positions in the instruction template; selecting the corresponding inference path constraint field content from the preset inference path library according to the job scenario type, and filling the content into the corresponding inference path constraint field position in the instruction template; and concatenating the filled instruction template to obtain the complete instruction text.

[0071] Taking a hoisting operation scenario as an example, the sample instruction text is as follows: The task description field is "Determine the violation of power safety regulations in the following image"; the prior condition field is "The current operation scenario is a hoisting operation, the image contains a crane boom and workers, and the known violation triggering condition is that there are personnel standing under the crane boom"; the reasoning path constraint field is "Please analyze in the following order: first, identify the position and coverage area of ​​the crane boom; second, identify the position of each worker; then, determine whether each worker is within the coverage area of ​​the crane boom; finally, combine the violation triggering condition to give the judgment result"; the output format constraint field is "The output should include three parts: scenario description information, reasoning process information, and violation judgment result." This example is only for illustrating the structure and content of the instruction text and does not constitute a limitation on the specific form of the instruction template.

[0072] Optionally, the terminal can also adaptively select a target instruction template from multiple preset candidate instruction templates based on the type of work scenario. Specifically, the terminal pre-stores multiple candidate instruction templates corresponding to different work scenario types. Each candidate instruction template has a different emphasis on the inference path constraint field. For example, for a hoisting operation scenario, it stores a first candidate instruction template that emphasizes spatial coverage relationship analysis and a second candidate instruction template that emphasizes personnel trajectory analysis. After obtaining the annotation information, the terminal selects the most matching candidate instruction template from the multiple candidate instruction templates as the target instruction template based on the characteristics of the target object's state in the annotation information (such as the number of workers, equipment posture, etc.). This adaptive selection mechanism enables the instruction text to match the specific scenario features, further improving the relevance of the results generated by the subsequent inference model.

[0073] Step S208: Input the target labeled image and instruction text into the preset inference model, and generate the inference process information of the target labeled image and the violation judgment result through the preset inference model according to the logical inference steps indicated by the instruction text.

[0074] The reasoning process information describes the reasoning process for determining the behavior of the labeled target image. For example, the reasoning process information is represented in structured text form, including scene description sub-information, key target confirmation sub-information, spatial relationship analysis sub-information, rule matching sub-information, and conclusion derivation sub-information.

[0075] The violation determination result is used to indicate whether there is a violation in the target labeled image and the specific type of violation, and is represented in the form of a binary determination value and a violation type label.

[0076] The pre-defined inference model refers to a multimodal large-scale model trained on general image and text data, based on a general multimodal large-scale model architecture, such as the Qwen-VL model or the LLaVA model. This pre-defined inference model has the ability to process both image and text inputs simultaneously, analyze and infer image content based on text instructions, and output text results that meet the instructions. Before deployment, the pre-defined inference model does not require additional training on data from power safety monitoring scenarios; it can complete inference tasks based solely on the general visual understanding and text reasoning capabilities learned from training on general data, according to the constraints of the instruction text.

[0077] In practice, the terminal simultaneously inputs the target labeled image and the instruction text into a preset inference model. The preset inference model first performs visual encoding on the target labeled image to obtain an image feature sequence; simultaneously, it performs text encoding on the instruction text to obtain a text feature sequence; then, it performs cross-modal fusion of the image feature sequence and the text feature sequence to obtain a fused feature sequence; finally, based on the fused feature sequence, it generates inference process information step by step according to the order specified by the inference path constraint field in the instruction text, and generates a violation judgment result based on the inference process information after the inference process information is generated.

[0078] Step S210: Combine the security monitoring images, instruction text, reasoning process information, and violation judgment results into multimodal sample data to construct a training dataset.

[0079] The training dataset is used to fine-tune the multimodal large model to be trained, resulting in a trained multimodal large model; the trained multimodal large model is used to identify violations in power operation scenarios.

[0080] The multimodal sample data refers to a data structure organized in the form of a four-tuple, denoted as D=(image, instruction, reasoning process, judgment result). Here, "image" corresponds to the original security monitoring image, "instruction" corresponds to the instruction text generated based on the annotation information and instruction template, "reasoning process" corresponds to the reasoning process information generated by the preset reasoning model, and "judgment result" corresponds to the violation judgment result generated by the preset reasoning model. The "image" in the multimodal sample data uses the original security monitoring image instead of the fused target annotation image, so that the downstream fine-tuned multimodal large model can directly perform reasoning based on the original image, without relying on the visual saliency enhancement introduced by the fusion processing.

[0081] In its implementation, the terminal encapsulates the security monitoring image, instruction text, reasoning process information, and violation judgment result into a single sample record according to the four-tuple structure of multimodal sample data. The terminal performs deduplication on multiple sample records, removing near-duplicate samples introduced during the keyframe extraction stage. The terminal then divides the deduplicated sample records into a training set, a validation set, and a test set according to a preset ratio; these three sets together constitute the training dataset. The training dataset is stored in a data format compatible with mainstream multimodal training frameworks (such as Qwen-VL and LLaVA), supporting transfer learning of subsequent large multimodal models on tasks such as hoisting operations and climbing ladders.

[0082] The aforementioned automated data construction method for multimodal chain-based inference in power safety monitoring scenarios involves target detection in safety monitoring images of power operation scenarios, followed by fusion of the detection results with the original images to obtain target-annotated images. This enhances the visual saliency of the target area, providing a structured prior knowledge of the target location for subsequent inference models. The method acquires annotation information containing three types of semantic anchors: operation scenario type, target object status, and violation triggering conditions. These are then transformed into instruction text with prior condition constraints and inference path constraints using a preset instruction template. This reduces annotation costs and inconsistencies compared to the traditional method of manually writing complete inference processes line by line. Finally, the target-annotated images and instruction text are input into a preset inference model, which then executes the inference according to the instructions. The logical reasoning steps of the instructions generate reasoning process information and violation judgment results containing complete reasoning paths. The originally implicit visual judgment process is explicitly decomposed into interpretable chain reasoning steps. Finally, the safety monitoring images, instruction text, reasoning process information and violation judgment results are combined into multimodal sample data and a training dataset is constructed. Each sample in the training dataset contains explicit chain reasoning information. This enables the multimodal large model obtained by fine-tuning based on the training dataset to improve from recognition based on appearance features to understanding and judgment based on reasoning process. In the power safety supervision scenario, this achieves robust understanding and interpretable judgment of multi-target spatial relationships, complex interactive behaviors and safety procedure logic, improving the accuracy of violation detection in power operation scenarios.

[0083] For the convenience of those skilled in the art, Figure 3 An exemplary flowchart of a dataset construction method is provided. Figure 3This paper demonstrates the complete construction process from power operation site monitoring videos to a multimodal training dataset. In practical applications, keyframes are first extracted from the video stream collected by video monitoring equipment deployed at the power construction site at a fixed frame rate, and blurred and duplicate frames are removed to obtain a safety monitoring image. This image is then input into a pre-trained target detection model for target detection, yielding detection results containing bounding boxes and category labels for the target objects. Simultaneously, annotators with safety supervision expertise provide three types of semantic anchor information—operation scenario type, target object status, and violation triggering conditions—as annotation information using annotation tools. Next, the safety monitoring image and the detection results are fused to obtain a target-annotated image. Finally, the annotation information is input into a preset instruction template to obtain an image with prior conditions. The instruction text for constraints and inference path constraints is then input into a preset inference model along with the target labeled image. The model generates inference process information and violation judgment results containing complete inference paths according to the logical inference steps indicated by the instruction text. Then, a dual-model consistency check is performed on the inference process information and violation judgment results. If the check passes, the security monitoring image, instruction text, inference process information, and violation judgment results are combined into multimodal sample data in the form of quadruples. If the check fails, the instruction template is adjusted and regenerated until the check passes. Finally, after deduplication, data augmentation, and sample balancing are performed on multiple multimodal sample data, a training dataset is constructed. This training dataset is used for supervised fine-tuning of the large multimodal model to be trained.

[0084] In another embodiment, the logical reasoning steps indicated by the instruction text include the following steps:

[0085] Identify the working environment and work type of the target labeled image; locate the target object corresponding to the work type in the working environment; analyze the positional relationship and interaction state between each target object to obtain the scene description information of the target labeled image; match the scene description information with the power safety regulations in the preset rule base to obtain the matching result; based on the matching result, generate the violation judgment result and generate the reasoning process information containing the logical reasoning path.

[0086] In practical applications, the logical reasoning steps indicated by the instruction text may include the following steps: scene semantic parsing, used to identify the working environment and work type of the target labeled image; key target confirmation, used to locate the target object corresponding to the work type in the working environment; spatial relationship reasoning, used to analyze the positional relationships and interaction states between various target objects to obtain scene description information of the target labeled image; safety procedure matching, used to match the scene description information with power safety procedures in the preset rule base to obtain matching results; and violation judgment, used to generate violation judgment results based on the matching results, as well as generate reasoning process information containing the logical reasoning path. The preset reasoning model executes the above steps sequentially according to the logical reasoning steps indicated by the instruction text.

[0087] The work environment refers to the overall characteristics of the physical work scene reflected in the target-annotated image, including the type of work location (e.g., indoor, outdoor, on a tower), the type of surrounding facilities (e.g., power transmission lines, substation equipment), and environmental conditions (e.g., lighting conditions, weather conditions). The work type refers to the specific work activity classification performed in this work environment, such as hoisting operations, working at heights, and live-line work.

[0088] In practice, the terminal analyzes the global visual features of the target labeled image through a preset inference model, and combines the job scenario type information provided by the prior condition field in the instruction text to output the job environment recognition result and the job type recognition result.

[0089] In practice, the terminal determines the set of key target object categories corresponding to the job type from a preset list of object categories based on the job type identification result; it then locates each target object belonging to the set of key target object categories in the target annotation image and records the position coordinates and category label of each target object. For example, when the job type identification result is a hoisting operation, the set of key target object categories is determined to include the boom, the object being hoisted, and the workers, and the target objects of the above categories are located in the target annotation image.

[0090] Among them, positional relationship refers to the relative positional relationship of each target object on the two-dimensional plane of the image, including left-right relationship, up-down relationship and containment relationship.

[0091] Interaction state refers to the interrelationship between target objects in the sense of physical operation, such as the "below" or "to the side" state between the operator and the boom, or the "on the ladder" or "below the ladder" state between the operator and the ladder.

[0092] The scene description information is represented in the form of structured text, which includes the identifiers and locations of each target object, as well as the positional relationships and interaction states between the target objects.

[0093] In practice, the scenario description information is matched against power safety regulations in a preset rule base to obtain a matching result. The preset rule base stores power safety regulation entries corresponding to each job type, with each entry represented as a triple of "job type—violation triggering condition—violation type". The terminal retrieves a set of power safety regulation entries from the preset rule base corresponding to the job type identification result; it then matches the positional relationships and interaction states in the scenario description information against the violation triggering conditions in that set of power safety regulation entries one by one; if the scenario description information contains content that satisfies a violation triggering condition, the matching result indicates that the power safety regulation entry has been matched; otherwise, the matching result indicates that the power safety regulation entry has not been matched.

[0094] In practice, a violation determination result is generated based on the violation type corresponding to the matched power safety regulation entries in the matching results; if no matched power safety regulation entries are found, the violation determination result indicates that there is no violation.

[0095] Simultaneously, the intermediate outputs of the above steps (scene semantic parsing, key target identification, spatial relationship reasoning, safety procedure matching, and violation judgment) are concatenated in chronological order to form reasoning process information containing the logical reasoning path. This reasoning process information is presented in a chain structure of "identification—location—analysis—matching—judgment," making each step of the reasoning process traceable.

[0096] The technical solution of this embodiment explicitly decomposes the reasoning process of the reasoning model into five steps: identification of the work environment and work type, location of the target object, analysis of positional relationship and interaction status, and generation of safety procedure matching and violation judgment results. This makes the originally implicit visual judgment process explicitly decomposed into multiple interpretable steps, constructing a complete reasoning chain of "target-relationship-rule-conclusion", which significantly improves its ability to understand complex spatial relationships, multi-target interactions and safety procedure logic.

[0097] In another embodiment, the positional relationships and interaction states between the target objects are analyzed to obtain scene description information of the target labeled image, including: extracting the two-dimensional visual features of each target object in the target labeled image; and based on the two-dimensional visual features, analyzing the relative distance, relative height hierarchy, and mutual occlusion relationship of each target object in three-dimensional space to obtain scene description information of the target labeled image.

[0098] Two-dimensional visual features refer to the features extracted from the two-dimensional pixel information of the target labeled image to describe the appearance and location of the target object. These features include the bounding box coordinates of the target object in the image, the aspect ratio of the bounding box, the visual feature vector of the target object, and the texture features of the region where the target object is located. The terminal extracts features from the region where each target object is located through the visual encoder in the preset inference model to obtain the two-dimensional visual feature vector corresponding to each target object.

[0099] The relative distance refers to the estimated distance of each target object relative to the camera in three-dimensional physical space. The terminal estimates the distance of each target object relative to the camera based on the bounding box size of each target object in the image, the vertical position of the bounding box in the image, and the preset camera intrinsic parameters.

[0100] Among them, relative height level refers to the height level of each target object relative to the ground plane in three-dimensional physical space. The terminal estimates the height level of each target object based on the vertical position of the lower edge of the bounding box in the image and the relative distance estimate.

[0101] Mutual occlusion relationship refers to the occlusion relationship between each target object in the image. The terminal determines whether a target object closer to the camera occludes a target object farther away from the camera based on the overlap between the bounding boxes of each target object and the relative distance estimate of each target object.

[0102] In practical applications, during hoisting operations, the terminal uses the aforementioned 3D spatial relationship analysis to distinguish between workers "close to the camera but actually far from the boom" and workers "directly below the boom." For workers whose bounding boxes in the 2D image are below the boom's bounding box but whose relative distance is greater than the boom's relative distance, the terminal determines that they are not directly below the boom, thus avoiding misclassifying such workers as being in the dangerous area below the boom. In high-altitude operations, the terminal uses 3D spatial relationship analysis to distinguish between workers "standing on a platform" and workers "climbing a ladder." Based on the relationship between the relative height levels of the worker's bounding box and the ladder's bounding box, the terminal determines whether the worker is actually on the ladder, thus avoiding misclassifying those standing on the platform as those climbing the ladder.

[0103] The technical solution of this embodiment analyzes the relative distance, relative height, and mutual occlusion relationships of each target object in three-dimensional space based on two-dimensional visual features. This enables the inference model to infer the three-dimensional spatial relationships of the target objects from two-dimensional images, overcoming the defect that general multimodal large models have difficulty accurately understanding the three-dimensional spatial relationships in two-dimensional images. By distinguishing between workers who are "close to the camera but actually far from the boom" and workers who are "directly below the boom", and by distinguishing between workers standing on the platform and workers climbing the ladder, misjudgments caused by two-dimensional visual information are avoided.

[0104] In another embodiment, before combining the security monitoring image, instruction text, reasoning process information, and violation determination result into multimodal sample data, a consistency check and self-feedback optimization operation is performed. Specifically, this includes: performing a consistency check on the reasoning process information and violation determination result to obtain a consistency check result; if the consistency check result indicates that the check has passed, performing the step of combining the security monitoring image, instruction text, reasoning process information, and violation determination result into multimodal sample data; if the consistency check result indicates that the check has failed, adjusting the instruction template, inputting the annotation information into the adjusted instruction template, and obtaining the updated instruction text; and regenerating the reasoning process information and violation determination result based on the target annotation image and the updated instruction text using a preset reasoning model until the consistency check is passed.

[0105] The consistency check is used to detect whether there are logical contradictions between the inference process information and the violation judgment result, and whether the inference process information and the violation judgment result are consistent with the semantic anchors in the annotation information. The consistency check result is represented in the form of a binary value, namely "passed the check" and "failed the check". The specific process of consistency check is described in the following embodiments and will not be elaborated here.

[0106] If the consistency verification result indicates that the verification is passed, the step of combining the security monitoring image, instruction text, reasoning process information and violation judgment result into multimodal sample data is performed. The specific process of this step is the same as the process of S210 in the aforementioned embodiment, and will not be described in detail here.

[0107] When the consistency check result indicates a failure, the terminal selects a target adjustment strategy from a set of preset template adjustment strategies based on the specific reason for the failure indicated in the consistency check result. The template adjustment strategies include three categories: constraint enhancement strategies, prompt structure change strategies, and multiple sampling strategies. The constraint enhancement strategy adds explicit constraints on the logical consistency between reasoning steps in the instruction template when the reason for failure is "a logical contradiction between the reasoning process information and the violation judgment result." The prompt structure change strategy adjusts the presentation order or expression of prior condition fields in the instruction template when the reason for failure is "inconsistency between the reasoning process information and the semantic anchor point." The multiple sampling strategy adds a command to the instruction template requiring the generation of multiple candidate reasoning processes when the reason for failure is "ambiguous expression of the reasoning process information." The terminal adjusts the instruction template according to the selected target adjustment strategy and inputs the annotation information into the adjusted instruction template to obtain the updated instruction text.

[0108] Then, based on the target labeled image and the updated instruction text, the pre-set inference model regenerates the inference process information and violation judgment result, and performs a consistency check on the regenerated inference process information and violation judgment result again until the consistency check result indicates that the check has passed.

[0109] The technical solution of this embodiment performs consistency verification on the inference process information and violation judgment results before combining multimodal sample data, and only combines samples that pass the verification into multimodal sample data, avoiding noise introduced by logical contradictions or inconsistencies with semantic anchors in the results generated by the inference model; by adjusting the instruction template and regenerating it when it fails the verification, a self-feedback optimization closed loop of "generation-verification-adjustment-regeneration" is formed, reducing the noise and error accumulation problems introduced during the automatic annotation process.

[0110] In another embodiment, a consistency check is performed on the reasoning process information and the violation judgment result to obtain a consistency check result, including: comparing the violation judgment result with a preset manually labeled answer using a preset check model to obtain a first check result; the first check result is used to indicate whether the violation judgment result is consistent with the manually labeled answer; comparing the reasoning process information with the violation judgment result using a preset reasoning model to obtain a second check result; the second check result is used to indicate whether the reasoning process information is consistent with the violation judgment result; if both the first check result and the second check result indicate consistency, the consistency check result is determined to be a pass check; if at least one check result in the first check result and the second check result indicates inconsistency, the consistency check result is determined to be a fail check.

[0111] Among them, the manual label answer refers to the manual labeling results that are pre-labeled by the labelers during the labeling stage to indicate whether there are violations in the security monitoring images and the specific types of violations.

[0112] The preset verification model is configured with a pre-written rule-based verification script for automatically comparing the violation judgment results with the manually labeled answers.

[0113] In specific implementation, a pre-defined verification model is used to compare the violation judgment result generated by the pre-defined inference model with the pre-defined manually labeled answer. If both the violation judgment value and the violation type label indicated by the violation judgment result are consistent with the manually labeled answer, the first verification result is determined to be consistent; otherwise, the first verification result is determined to be inconsistent. Optionally, the first verification result can be represented in the form of a binary value, namely "consistent" and "inconsistent".

[0114] Then, the reasoning process information generated by the preset reasoning model and the violation judgment result can be input into the preset reasoning model. The preset reasoning model then determines whether the reasoning process described by the reasoning process information is consistent with the conclusion indicated by the violation judgment result. This process can specifically include: the preset reasoning model extracting key reasoning nodes from the reasoning process information; the preset reasoning model extracting conclusive statements from the violation judgment result; and the preset reasoning model determining whether the logical conclusions corresponding to the key reasoning nodes are consistent with the conclusive statements. Optionally, the second verification result can be represented in the form of binary values, namely "consistent" and "inconsistent".

[0115] In specific implementation, if both the first and second verification results are consistent, the consistency verification result is determined to be passed; if either the first or second verification result is inconsistent, the consistency verification result is determined to be failed. This method ensures consistency between the violation judgment result and the manually labeled answer through a verification model, and also ensures consistency between the reasoning process information and the violation judgment result through a pre-set reasoning model. Only samples with correct answers and reasonable reasoning are retained in the training dataset, reducing noise and errors introduced during automatic labeling.

[0116] The technical solution of this embodiment compares the violation judgment result with the manually labeled answer to obtain the first verification result through a preset verification model, and performs consistency verification between the reasoning process information and the violation judgment result through a preset reasoning model to obtain the second verification result. The consistency of both is used as the basis for determining the consistency verification result. Only samples with correct answers and reasonable reasoning are retained in the training dataset, thereby ensuring the quality of automatically constructed reasoning data in terms of both answer correctness and reasoning logic correctness, and reducing the noise accumulation problem introduced in the automatic labeling process.

[0117] In another embodiment, after combining the safety monitoring images, instruction text, reasoning process information, and violation judgment results into multimodal sample data to construct a training dataset, the method further includes: fine-tuning the multimodal large model to be trained using the training dataset to obtain a trained multimodal large model; inputting the power operation scene image to be detected and the behavior detection instruction text into the trained multimodal large model to obtain the violation judgment result for the power operation scene image, and the reasoning process information corresponding to the violation judgment result.

[0118] The multimodal large model to be trained refers to a multimodal large model that has been pre-trained on general image and text data but has not yet been trained on power safety monitoring scenario data. The multimodal large model that has been trained refers to a multimodal large model that has been fine-tuned on the training dataset and is capable of identifying and reasoning about violations in power safety monitoring scenarios. The fine-tuning adopts the supervised fine-tuning (SFT) method, that is, using multimodal sample data in the training dataset as supervision samples, using safety monitoring images and instruction text in the multimodal sample data as model inputs, and using the reasoning process information and violation judgment results in the multimodal sample data as the model target output. The parameters of the multimodal large model to be trained are updated by minimizing the difference between the model's predicted output and the target output.

[0119] For example, during fine-tuning, the terminal employs an autoregressive language modeling loss function. This function weights and sums the token-level generation losses for inference process information and violation judgment results to obtain the total loss. The weights corresponding to inference process information are higher than those corresponding to violation judgment results, ensuring the model focuses more on learning the inference process during fine-tuning. For training hyperparameter selection, the terminal uses preset learning rates, batch sizes, and training epochs. The specific values ​​are chosen based on the size of the training dataset and the number of parameters in the large multimodal model to be trained.

[0120] Among them, the power operation scene images to be detected refer to power operation scene images that are collected in real time during the model application stage and need to be identified by the trained multimodal large model for violation recognition.

[0121] The behavior detection instruction text refers to the text instructions used to direct the trained multimodal large-scale model to perform the violation detection task. Its structure is consistent with the instruction text structure used in the training phase, but it does not contain specific prior violation triggering conditions, only task descriptions and output format constraints. The terminal inputs the power operation scene image to be detected and the behavior detection instruction text into the trained multimodal large-scale model, which then autonomously generates inference process information and violation judgment results based on the image content.

[0122] The technical solution of this embodiment utilizes the training dataset to perform supervised fine-tuning of the multimodal large model to be trained, thereby improving the multimodal large model from recognition based on appearance features to understanding and judgment based on reasoning processes, significantly enhancing its ability to recognize complex behaviors and generalize in power safety supervision scenarios.

[0123] For the convenience of those skilled in the art, Figure 4 An exemplary diagram illustrating the input and output of a multimodal large model during the inference phase is provided. Figure 4 This demonstrates the input and output formats of a pre-defined inference model during the data construction phase and the application phase of a trained multimodal large model. In practical applications, the input consists of two parts: a target-annotated image and instruction text. The target-annotated image is an image overlaid with the target object's bounding box and category label on the original safety monitoring image. The target object bounding box is drawn with a preset color and line width, and the corresponding category label is marked near the bounding box, thereby visually enhancing the salience of the areas where core targets such as workers, cranes, and ladders are located. The instruction text is composed of task description fields, prior condition fields, inference path constraint fields, and output format constraint fields. It clearly defines the prior facts such as the current work scenario type, target object status, and violation triggering conditions, and specifies the analysis steps and output format that the model should follow. The output consists of... The system comprises two parts: reasoning process information and violation determination results. The reasoning process information is presented in structured text format, unfolding step by step in the order of scene semantic parsing, key target confirmation, spatial relationship reasoning, safety procedure matching, and violation behavior determination. Each step provides the corresponding target object identifier, location relationship description, and rule matching basis. The violation determination results provide the final safety conclusion in the form of a binary judgment value and a violation type label. Simultaneously, the target object identifier and spatial relationship description corresponding to the key reasoning nodes are parsed from the reasoning process information. Visual annotations such as bounding boxes, relationship indicators, and distance annotations are added to the target object area corresponding to the target object identifier on the target annotation image, enabling monitoring personnel to obtain both the violation determination results and the corresponding reasoning basis at the same time, facilitating the review and decision-making of the determination results.

[0124] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0125] Based on the same inventive concept, this application also provides an automated construction device for multimodal chain inference data in power safety monitoring scenarios, which implements the aforementioned automated construction method for multimodal chain inference data in power safety monitoring scenarios. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the automated construction device for multimodal chain inference data in power safety monitoring scenarios provided below can be found in the limitations of the automated construction method for multimodal chain inference data in power safety monitoring scenarios described above, and will not be repeated here.

[0126] In one exemplary embodiment, such as Figure 5 As shown, a multimodal chain-based inference data automated construction device for power safety monitoring scenarios is provided, comprising:

[0127] The detection module 510 is used to perform target detection on safety monitoring images in power operation scenarios to obtain the detection results of the safety monitoring images; the detection results are used to mark the target objects in the safety monitoring images.

[0128] The acquisition module 520 is used to acquire the annotation information corresponding to the safety monitoring image; the annotation information includes semantic information for indicating the type of work scenario, the status of the target object and the conditions for triggering violations;

[0129] The processing module 530 is used to fuse the security monitoring image with the detection result to obtain a target annotation image, and to input the annotation information into a preset instruction template to obtain instruction text;

[0130] The determination module 540 is used to input the target labeled image and the instruction text into a preset reasoning model, and generate reasoning process information and violation determination result of the target labeled image through the preset reasoning model according to the logical reasoning steps indicated by the instruction text; the reasoning process information is used to describe the reasoning process of determining the behavior of the target labeled image;

[0131] The construction module 550 is used to combine the safety monitoring image, the instruction text, the reasoning process information and the violation judgment result into multimodal sample data to construct a training dataset; the training dataset is used to fine-tune the multimodal large model to be trained to obtain a trained multimodal large model; the trained multimodal large model is used to identify violations in power operation scenarios.

[0132] In one embodiment, the determination module 540 is specifically used to identify the working environment and working type of the target labeled image; locate the target object corresponding to the working type in the working environment; analyze the positional relationship and interaction state between the target objects to obtain the scene description information of the target labeled image; match the scene description information with the power safety regulations in the preset rule base to obtain the matching result; and generate the violation determination result based on the matching result, as well as the reasoning process information containing the logical reasoning path.

[0133] In one embodiment, the determination module 540 is specifically used to extract the two-dimensional visual features of each target object in the target annotation image; based on the two-dimensional visual features, analyze the relative distance, relative height hierarchy and mutual occlusion relationship of each target object in three-dimensional space, and obtain the scene description information of the target annotation image.

[0134] In one embodiment, the construction module 550 is specifically used to perform consistency verification on the inference process information and the violation determination result to obtain a consistency verification result; if the consistency verification result indicates that the verification is passed, the step of combining the security monitoring image, the instruction text, the inference process information, and the violation determination result into multimodal sample data is executed; if the consistency verification result indicates that the verification is failed, the instruction template is adjusted, and the annotation information is input into the adjusted instruction template to obtain an updated instruction text; based on the target annotation image and the updated instruction text, the inference process information and the violation determination result are regenerated using the preset inference model until the consistency verification is passed.

[0135] In one embodiment, the construction module 550 is specifically used to compare the violation judgment result with the preset manually labeled answer using a preset verification model to obtain a first verification result; the first verification result is used to indicate whether the violation judgment result is consistent with the manually labeled answer; the reasoning process information is compared with the violation judgment result using the preset reasoning model to obtain a second verification result; the second verification result is used to indicate whether the reasoning process information is consistent with the violation judgment result; if both the first verification result and the second verification result indicate consistency, the consistency verification result is determined to be a pass verification; if at least one of the first verification result and the second verification result indicates inconsistency, the consistency verification result is determined to be a fail verification.

[0136] In one embodiment, the automated construction device for multimodal chain-based inference data further includes an inference module, which is specifically used to fine-tune the multimodal large model to be trained using the training dataset to obtain a trained multimodal large model; input the power operation scene image to be detected and the behavior detection instruction text into the trained multimodal large model to obtain the violation judgment result for the power operation scene image, and the inference process information corresponding to the violation judgment result.

[0137] The modules in the aforementioned automated construction device for multimodal chain-based inference data for power safety monitoring scenarios can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0138] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a multimodal chain-based inference data automated construction method for power safety monitoring scenarios. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0139] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0140] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0141] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0142] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0143] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0144] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0145] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0146] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for automatically constructing multimodal chain-based inference data for power safety monitoring scenarios, characterized in that, The method includes: Target detection is performed on safety monitoring images in power operation scenarios to obtain detection results for the safety monitoring images; the detection results are used to label target objects in the safety monitoring images. Obtain the annotation information corresponding to the safety monitoring image; the annotation information includes semantic information used to indicate the type of work scenario, the status of the target object, and the conditions for triggering violations; The security monitoring image and the detection result are fused to obtain a target labeled image, and the labeled information is input into a preset instruction template to obtain instruction text; The target labeled image and the instruction text are input into a preset inference model. The preset inference model generates inference process information and violation judgment result of the target labeled image according to the logical inference steps indicated by the instruction text. The inference process information is used to describe the inference process of behavior judgment of the target labeled image. The safety monitoring images, the instruction text, the reasoning process information, and the violation determination results are combined into multimodal sample data to construct a training dataset. The training dataset is used to fine-tune the multimodal large model to be trained, resulting in a trained multimodal large model. The trained multimodal large model is used to identify violations in power operation scenarios.

2. The method according to claim 1, characterized in that, The logical reasoning steps indicated by the instruction text include the following steps: Identify the working environment and job type of the target labeled image; locate the target object corresponding to the job type within the working environment; Analyze the positional relationships and interaction states between the target objects to obtain scene description information of the target labeled image; The scenario description information is matched with the power safety regulations in the preset rule base to obtain the matching result; Based on the matching result, the violation determination result is generated, as well as the reasoning process information containing the logical reasoning path is generated.

3. The method according to claim 2, characterized in that, The analysis of the positional relationships and interaction states between the target objects to obtain scene description information of the target-annotated image includes: Extract the two-dimensional visual features of each target object in the target labeled image; Based on the two-dimensional visual features, the relative distance, relative height, and mutual occlusion relationships of each target object in three-dimensional space are analyzed to obtain scene description information of the target labeled image.

4. The method according to claim 1, characterized in that, Before combining the security monitoring image, the instruction text, the reasoning process information, and the violation determination result into multimodal sample data, the method further includes: The consistency verification is performed on the reasoning process information and the violation determination result to obtain the consistency verification result; If the consistency verification result indicates that the verification has passed, the step of combining the security monitoring image, the instruction text, the reasoning process information, and the violation determination result into multimodal sample data is performed. If the consistency check result indicates that the check fails, adjust the instruction template, input the annotation information into the adjusted instruction template, and obtain the updated instruction text; The preset inference model regenerates the inference process information and the violation judgment result based on the target labeled image and the updated instruction text, until the consistency check is passed.

5. The method according to claim 4, characterized in that, The process of performing a consistency check on the reasoning process information and the violation determination result to obtain a consistency check result includes: The violation determination result is compared with the preset manually labeled answer using a preset verification model to obtain a first verification result; the first verification result is used to indicate whether the violation determination result is consistent with the manually labeled answer. The reasoning process information is compared with the violation determination result using the preset reasoning model to obtain a second verification result; the second verification result is used to indicate whether the reasoning process information is consistent with the violation determination result. If both the first verification result and the second verification result indicate consistency, the consistency verification result is determined to be a pass verification. If at least one of the verification results in the first verification result and the second verification result is inconsistent, the consistency verification result is determined to be a failure.

6. The method according to claim 1, characterized in that, After combining the security monitoring image, the instruction text, the reasoning process information, and the violation determination result into multimodal sample data to construct a training dataset, the method further includes: The multimodal large model to be trained is fine-tuned using the training dataset to obtain the trained multimodal large model. The image of the power operation scene to be detected and the text of the behavior detection instruction are input into the trained multimodal large model to obtain the violation judgment result for the power operation scene image, as well as the inference process information corresponding to the violation judgment result.

7. An automated data construction device for multimodal chain-based inference in power safety monitoring scenarios, characterized in that, The device includes: The detection module is used to perform target detection on safety monitoring images in power operation scenarios to obtain the detection results of the safety monitoring images; the detection results are used to mark the target objects in the safety monitoring images. The acquisition module is used to acquire the annotation information corresponding to the safety monitoring image; the annotation information includes semantic information indicating the type of operation scenario, the status of the target object, and the conditions for triggering violations; The processing module is used to fuse the security monitoring image with the detection result to obtain a target labeled image, and to input the labeled information into a preset instruction template to obtain instruction text; The judgment module is used to input the target labeled image and the instruction text into a preset inference model, and generate inference process information and violation judgment result of the target labeled image through the preset inference model according to the logical inference steps indicated by the instruction text; the inference process information is used to describe the inference process of judging the behavior of the target labeled image; The construction module is used to combine the safety monitoring image, the instruction text, the reasoning process information, and the violation judgment result into multimodal sample data to construct a training dataset; the training dataset is used to fine-tune the multimodal large model to be trained to obtain a trained multimodal large model; the trained multimodal large model is used to identify violations in power operation scenarios.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images