Multi-modal behavior detection method, device and computer equipment of dual data enhancement and implicit inference adapting to power scenario

Abstract

The application relates to a multi-modal behavior detection method, device and computer equipment for dual data enhancement and implicit inference adapting to a power scene. The method comprises the following steps: acquiring a first work image of a power work scene, the first work image comprising an image in which a target object is in a compliant operation behavior; generating a second work image based on the first work image by adopting a preset data generation strategy, and constructing a sample data set according to the first work image and the second work image; the data generation strategy comprises local modification based on image content and image generation based on semantic description; the second work image comprises an image in which the target object is in a non-compliant operation behavior; generating structured rule knowledge for the power work scene; training a multi-modal large model to be trained according to the sample data set and the structured rule knowledge, and obtaining a trained multi-modal large model. The method can improve the accuracy of behavior detection of work personnel in a power safety monitoring scene.

Description

Technical Field

[0001] This application relates to the field of power safety supervision technology, and in particular to a multimodal behavior detection method, device, computer equipment, computer-readable storage medium and computer program product that are adapted to power scenarios and employ dual data augmentation and implicit reasoning. Background Technology

[0002] In the safety supervision of power construction and operation and maintenance, the automatic identification of whether workers are wearing safety belts and other protective equipment in accordance with regulations is a key link in realizing intelligent safety supervision and unmanned inspection. With the development of computer vision technology, image analysis-based safety detection methods are gradually being applied, aiming to replace manual inspections and improve supervision efficiency and coverage.

[0003] Currently, traditional safety detection methods are typically based on object detection or image classification models. These models are trained on a large number of labeled images to learn the visual features of "wearing a seatbelt" and "not wearing a seatbelt," and then perform binary classification on the input image. However, such methods still face significant shortcomings in actual power operation environments: Firstly, violations (such as not wearing a seatbelt) rarely occur in high-risk work scenarios, making it difficult to collect real negative samples. This leads to a severe imbalance between positive and negative samples in the training data, causing the model to easily develop a prediction bias towards positive samples after training, resulting in insufficient sensitivity to identify rare violations. Secondly, the rules for determining seatbelt wearing are highly dependent on the specific work scenario, and the models of such methods typically lack a unified expression and adaptive ability for complex judgment logic in different scenarios. This makes them prone to misjudgment or missed judgment in variable work environments, making it difficult to meet the actual power safety supervision requirements in terms of behavior detection accuracy. Summary of the Invention

[0004] Therefore, it is necessary to address the aforementioned technical problems by providing a multimodal behavior detection method, device, computer equipment, computer-readable storage medium, and computer program product that adapts to the power scenario and employs dual data augmentation and implicit reasoning to improve the accuracy of detecting worker behavior in power safety monitoring scenarios.

[0005] Firstly, this application provides a multimodal behavior detection method adapted to power scenarios, incorporating dual data augmentation and implicit reasoning, including:

[0006] Acquire a first operational image of a power operation scenario, wherein the first operational image includes an image of the target object performing compliant operational behavior;

[0007] A second task image is generated based on the first task image using a preset data generation strategy, and a sample dataset is constructed based on the first task image and the second task image; the data generation strategy includes local modification based on image content and image generation based on semantic description; the second task image includes an image of a target object performing an illegal operation.

[0008] Generate structured rule knowledge for the power operation scenario; the structured rule knowledge includes the mapping relationship between operation scenario characteristics, safe operation rules, and violation judgment results;

[0009] The multimodal large model to be trained is trained based on the sample dataset and the structured rule knowledge to obtain the trained multimodal large model; the multimodal large model is used to output the corresponding behavior detection results based on the job image to be detected.

[0010] In one embodiment, the step of generating a second job image based on the first job image using a preset data generation strategy includes:

[0011] The safety protection area in the first operation image is identified by a preset semantic segmentation model;

[0012] Under the constraints of maintaining the posture of the target object and the spatial relationship between the scene structure of the power operation scenario and the first operation image, the protective equipment object in the safety protection area is removed by a preset image editing model to obtain the second operation image.

[0013] In one embodiment, the step of generating a second job image based on the first job image using a preset data generation strategy includes:

[0014] The first task image is semantically parsed using a pre-defined visual model to generate descriptive text for the first task image; the descriptive text includes at least the environmental features, target object pose, and scene context information of the first task image.

[0015] A second operational image, excluding protective equipment objects, is generated based on the descriptive text using a preset text-to-image generation model.

[0016] In one embodiment, constructing a sample dataset based on the first job image and the second job image includes:

[0017] Image enhancement operations are performed on the first job image and the second job image respectively; the image enhancement operations include one or more of the following: rotation, scaling, flipping, brightness perturbation, and noise injection.

[0018] Based on each of the first job images and each of the second job images after the image enhancement operation, the sample dataset is constructed; the ratio of the number of the first job images and the number of the second job images in the sample dataset reaches a preset ratio.

[0019] In one embodiment, training the multimodal large model to be trained based on the sample dataset and the structured rule knowledge to obtain the trained multimodal large model includes:

[0020] The structured rule knowledge is transformed into structured prompt text;

[0021] The sample dataset and the corresponding structured prompt text are used as input, and the corresponding violation judgment result is used as the expected output. The multimodal large model to be trained is fine-tuned so that the multimodal large model to be trained establishes an alignment relationship between visual features, scene semantics and rule logic, and the trained multimodal large model is obtained.

[0022] In one embodiment, after training the multimodal large model to be trained based on the sample dataset and the structured rule knowledge to obtain the trained multimodal large model, the method further includes:

[0023] The image of the task to be detected is input into the trained multimodal large model;

[0024] The trained multimodal large model extracts the visual features and scene semantics of the work image to be detected, and activates safety operation rules that match the work scene features of the work image to be detected based on the alignment relationship, and outputs the behavior detection result of whether the work image to be detected belongs to a violation of the operation behavior.

[0025] Secondly, this application also provides a multimodal behavior detection device adapted to power scenarios with dual data augmentation and implicit reasoning, including:

[0026] The acquisition module is used to acquire a first operation image of a power operation scenario, wherein the first operation image includes an image of a target object performing compliant operation behavior;

[0027] A construction module is used to generate a second task image based on the first task image using a preset data generation strategy, and to construct a sample dataset based on the first task image and the second task image; the data generation strategy includes local modification based on image content and image generation based on semantic description; the second task image includes an image of a target object performing an illegal operation.

[0028] The generation module is used to generate structured rule knowledge for the power operation scenario; the structured rule knowledge includes the mapping relationship between operation scenario characteristics, safe operation rules, and violation judgment results;

[0029] The training module is used to train the multimodal large model to be trained based on the sample dataset and the structured rule knowledge, so as to obtain the trained multimodal large model; the multimodal large model is used to output the corresponding behavior detection results based on the task image to be detected.

[0030] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0031] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0032] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0033] The aforementioned multimodal behavior detection method, device, computer equipment, computer-readable storage medium, and computer program product adapted to power scenarios with dual data augmentation and implicit inference first acquires a first operation image of the power operation scenario. This first operation image includes an image of the target object performing compliant operations. Then, a second operation image is generated based on the first operation image using a preset data generation strategy. This dual data generation strategy, consisting of local modification of image content and image generation based on semantic description, transforms scarce compliant samples into high-quality negative samples of non-compliant behavior, effectively expanding the data coverage of rare non-compliant scenarios. It constructs a training dataset with balanced positive and negative samples, fundamentally alleviating the model's prediction bias towards positive samples and significantly improving the model's sensitivity and accuracy in identifying non-compliant behavior. Simultaneously, through… Structured rule knowledge, encompassing the mapping relationships between work scenario features, safe operating rules, and violation judgment results, is generated. This rule knowledge, along with a balanced sample dataset, is used to train a multimodal large-scale model. This enables the model to simultaneously learn visual features and complex judgment logic within a unified framework, establishing an intrinsic connection between scenarios and rules. Consequently, it possesses adaptive reasoning capabilities for multiple scenarios and conditions, accurately invoking appropriate judgment logic when facing different work environments. This effectively avoids misjudgments or omissions caused by the complexity and variability of rules. Finally, the trained multimodal large-scale model can directly output accurate behavior detection results based on the work images to be detected, improving the accuracy, robustness, and generalization ability of worker behavior detection in power safety supervision scenarios. This meets the reliability requirements of intelligent safety supervision systems in complex actual work environments. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is an application environment diagram of a multimodal behavior detection method with dual data augmentation and implicit reasoning adapted to power scenarios, as shown in one embodiment.

[0036] Figure 2 This is a flowchart illustrating a multimodal behavior detection method with dual data augmentation and implicit reasoning adapted to a power scenario, as shown in one embodiment.

[0037] Figure 3 This is a flowchart of a multimodal behavior detection method with dual data augmentation and implicit reasoning adapted to power scenarios in one embodiment.

[0038] Figure 4 This is a schematic diagram of the input and output of a multimodal large model in one embodiment;

[0039] Figure 5 This is a structural block diagram of a multimodal behavior detection device with dual data augmentation and implicit reasoning adapted to a power scenario, as shown in one embodiment.

[0040] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0042] The multimodal behavior detection method with dual data augmentation and implicit reasoning adapted to power scenarios provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.

[0043] Terminal 102 acquires a first operation image of a power operation scenario, which includes an image of the target object performing compliant operation behavior. Terminal 102 generates a second operation image based on the first operation image using a preset data generation strategy, and constructs a sample dataset based on the first and second operation images. The data generation strategy includes local modification based on image content and image generation based on semantic description. The second operation image includes an image of the target object performing non-compliant operation behavior. Terminal 102 generates structured rule knowledge for the power operation scenario. The structured rule knowledge includes the mapping relationship between operation scenario features, safe operation rules, and violation judgment results. Terminal 102 trains the multimodal large model to be trained based on the sample dataset and the structured rule knowledge to obtain the trained multimodal large model. The multimodal large model is used to output the corresponding behavior detection results based on the operation image to be detected.

[0044] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0045] In one exemplary embodiment, such as Figure 2 As shown, a multimodal behavior detection method with dual data augmentation and implicit reasoning adapted to power scenarios is provided, and this method is applied to... Figure 1 Taking terminal 102 as an example, the explanation includes:

[0046] Step S202: Obtain the first operation image of the power operation scenario.

[0047] The first operational image includes images of the target object performing compliant operational behavior.

[0048] The target group may include personnel who perform power construction, power operation and maintenance or power repair tasks in power operation scenarios, such as workers who perform high-altitude work, live-line work or power outage maintenance work in scenarios such as power poles, transmission towers, substations, valve halls or liftable work platforms.

[0049] Among them, compliant operation behavior refers to the behavior of the target object to correctly wear safety protective equipment and perform standardized actions in the corresponding work scenario in accordance with the requirements of the power safety work procedures. Safety protective equipment includes safety belts, safety helmets, insulating gloves, insulating shoes or fall arresters and other protective equipment.

[0050] The first operational image is image data obtained by the terminal through image acquisition equipment deployed in the power operation scenario, which records the target object in a compliant operation state. This image data can be video frames collected in real time by an industrial camera, surveillance camera or camera module carried by an inspection robot, or image frames extracted from historical surveillance recordings.

[0051] Step S204: Generate a second job image based on the first job image using a preset data generation strategy, and construct a sample dataset based on the first job image and the second job image.

[0052] The data generation strategies include local modification based on image content and image generation based on semantic description. In other words, the data generation strategies include two strategies: local modification based on image content and image generation based on semantic description. These two strategies constitute a dual data augmentation mechanism.

[0053] Among them, local modification based on image content refers to the terminal selectively editing the area of ​​the safety protection equipment worn by the target object in the image while keeping the overall scene structure of the first operation image unchanged, and converting the compliant operation behavior image into the non-compliant operation behavior image by removing the safety protection equipment.

[0054] Among them, image generation based on semantic description refers to the terminal obtaining descriptive text by performing semantic parsing on the first operation image, and then regenerating an image of the violation behavior without safety protection equipment based on the descriptive text.

[0055] The second type of operational image includes images of the target object engaging in unauthorized operations. Unauthorized operations refer to the target object's failure to wear safety protective equipment as required by the electrical safety work regulations in the corresponding operational scenario, such as not wearing a safety belt, not wearing a safety helmet, or not wearing insulating protective gear.

[0056] The sample dataset consists of a first task image and a second task image. The first task image is used as a positive sample, and the second task image is used as a negative sample. The label corresponding to the positive sample is compliance, and the label corresponding to the negative sample is violation.

[0057] In practice, the terminal selects compliant samples from the first operational image as the processing object, and generates corresponding candidate sets of second operational images according to the image content-based local modification strategy and the semantic description-based image generation strategy, respectively. The images in the candidate sets are then subjected to quality screening, and the screened images are used as the final second operational images. After generating the second operational images, the terminal stores the first and second operational images according to a preset data organization format, and labels each image with the corresponding scene type, operational height, protection conditions, and compliance or non-compliance tags, forming a structured sample dataset.

[0058] In practical applications, by employing two data generation strategies—one based on local image content modification and the other based on semantic description—a sufficient number of compliant samples can be transformed into non-compliant samples. Since the non-compliant samples generated by the image content modification strategy retain the physical authenticity of the original scene, while the non-compliant samples generated by the semantic description strategy have scene diversity, the two strategies complement each other, enabling the generated non-compliant samples to meet training requirements in terms of both authenticity and diversity. This alleviates the problem of scarce non-compliant samples and imbalanced distribution of positive and negative samples in power operation scenarios from the source, providing a data foundation for subsequent model training.

[0059] Step S206: Generate structured rule knowledge for power operation scenarios.

[0060] Among them, structured rule knowledge includes the mapping relationship between work scenario characteristics, safe operation rules, and violation judgment results.

[0061] Among them, the operation scenario characteristics refer to the characteristic information used to characterize the attributes of the power operation scenario, including the scenario type, operation height, protection conditions and operation location, etc. The scenario type refers to the type of operation such as power pole operation, transmission tower operation, slope operation, valve hall operation or liftable work platform operation; the operation height refers to the vertical distance of the target object from the ground; the protection conditions refer to whether there are protective facilities such as protective fences, safety nets or fall arresters in the operation scenario; the operation location refers to the specific location of the target object, such as on the platform, ground, fence or tower body.

[0062] Among them, the safety operation rules refer to the requirements for wearing safety protective equipment set for different work scenarios. The safety operation rules are characterized by conditionality and multiple branches, that is, under the same work scenario, different wearing requirements are corresponding to different combinations of working height, protection conditions or working position.

[0063] Among them, the violation determination result refers to the judgment conclusion that indicates whether the target object has violated the rules, which is derived from the characteristics of the work scenario and the safety operation rules. The violation determination result includes two states: violation and compliance.

[0064] In specific implementation, the process of generating structured rule knowledge at the terminal includes: extracting the original rule text from the power safety work procedure document, performing natural language parsing on the original rule text, identifying the scenario conditions, height thresholds, protection conditions, and corresponding wearing requirements, and structuring the identified information according to the logical link of work scenario characteristics, safe operation rules, and violation judgment results to form a mapping table; the rule that safety belts must be worn when climbing mobile manned platforms, transmission towers, utility poles, movable wooden ladders, scaffolding, etc., at heights exceeding 2 meters, is structured into a mapping entry for scenario type: utility pole work, transmission tower work, or scaffolding work; work height: greater than 2 meters; protection condition: no protective fence; safe operation rule: safety belts must be worn; violation judgment result: not wearing a safety belt is a violation; the rule that safety belt wearing must be checked when there is no protective fence in a slope work scenario is structured into a mapping entry for scenario type: slope. The following are mapping entries: The operation and protection conditions are no protective fence; the safety operation rule is that a safety belt must be worn; and the violation judgment result is that not wearing a safety belt constitutes a violation. The rule that safety belt wearing must be checked on a liftable work platform in a valve hall operation scenario, but not on the ground, is structured as follows: Scenario type: valve hall operation; Work location: liftable work platform or ground; Safety operation rule: safety belt must be worn on the platform, but not on the ground; Violation judgment result: not wearing a safety belt on the platform constitutes a violation, but not on the ground. Similarly, the rule that safety belt wearing must be checked when standing on a fenced relay box work platform, but not when not standing on the fence, is structured as follows: Scenario type: relay box work platform operation; Protection conditions: fenced; Work location: on or off the fence; Safety operation rule: safety belt must be worn on the fence, but not on the off-fence; Violation judgment result: not wearing a safety belt on the fence constitutes a violation, but not on the off-fence.

[0065] In practical applications, by uniformly mapping the empirical rules in the power safety work procedures to the mapping relationship between work scenario characteristics, safe operation rules and violation judgment results, the scattered, conditional and multi-branch safety procedures are transformed into a computable semantic structure. This enables the rule information to be injected in a structured form during subsequent model training, providing a knowledge base for the model to establish the intrinsic relationship between scenarios and rules, thereby supporting the model to perform adaptive reasoning under multiple scenarios and multiple rule conditions.

[0066] Step S208: Train the multimodal large model to be trained based on the sample dataset and structured rule knowledge to obtain the trained multimodal large model.

[0067] Among them, the multimodal large model is used to output the corresponding behavior detection results based on the image of the task to be detected.

[0068] Among them, the multimodal large model to be trained refers to a large model that has the ability to process image and text modal information at the same time and has not yet been fine-tuned for the behavior detection task in the power operation scenario. The multimodal large model to be trained adopts a visual language model architecture, that is, a large model that uses a pre-trained visual encoder to extract image features and a pre-trained language decoder to process text input and generate output.

[0069] Among them, the trained multimodal large model refers to a large model that has been fine-tuned for the behavior detection task in power operation scenarios and can directly output behavior detection results based on the input operation images. The behavior detection results include two judgment conclusions: violation and compliance, as well as the corresponding scene features, activated safety operation rules, and judgment criteria.

[0070] In specific implementation, the process of training the multimodal large model to be trained by the terminal based on the sample dataset and structured rule knowledge includes: associating the images in the sample dataset with the structured rule knowledge to construct training sample pairs. Each training sample pair includes the task image input, the structured rule knowledge input, and the corresponding violation judgment result label; the terminal inputs the training sample pairs into the multimodal large model to be trained, and through model fine-tuning, enables the model to learn the complete reasoning path from image input to scene recognition, then to rule matching and final judgment. During the training process, the difference between the model output and the label is calculated through the loss function, and the model parameters are updated through backpropagation until the model converges, and the trained multimodal large model is obtained.

[0071] In practical applications, by using sample datasets and structured rule knowledge together for training multimodal large models, the models can learn visual features and complex judgment logic synchronously within a unified framework, establishing an intrinsic connection between scenes and rules. This enables the models to have adaptive reasoning capabilities for multiple scenes and multiple condition combinations, allowing them to directly output behavior detection results based on the image of the operation to be detected during subsequent reasoning processes. This improves the accuracy and generalization ability of behavior detection in power safety supervision scenarios.

[0072] In the aforementioned multimodal behavior detection method adapted to power scenarios using dual data augmentation and implicit inference, a first operation image of the power operation scenario is first acquired. This first operation image includes images of the target object performing compliant operations. Then, a second operation image is generated based on the first operation image using a preset data generation strategy. This dual data generation strategy, consisting of local modification of image content and image generation based on semantic description, transforms scarce compliant samples into high-quality negative samples of violations, effectively expanding the data coverage of rare violation scenarios and constructing a training dataset with balanced positive and negative samples. This fundamentally alleviates the model's prediction bias towards positive samples and significantly improves the model's sensitivity and accuracy in identifying violations. Simultaneously, by generating images containing operation scenario features and safety... The structured rule knowledge mapping the relationship between operating rules and violation judgment results is used together with a balanced sample dataset for training a multimodal large model. This enables the model to learn visual features and complex judgment logic simultaneously within a unified framework, establishing an intrinsic connection between scenarios and rules. This results in adaptive reasoning capabilities for multiple scenarios and conditions, accurately invoking the corresponding judgment logic when facing different operating environments, effectively avoiding misjudgments or omissions caused by complex and ever-changing rules. Finally, the trained multimodal large model can directly output accurate behavior detection results based on the image of the operation to be detected, improving the accuracy, robustness, and generalization ability of operator behavior detection in power safety supervision scenarios, and meeting the reliability requirements of intelligent safety supervision systems in actual complex operating environments.

[0073] For the convenience of those skilled in the art, Figure 3 This paper provides an exemplary architecture diagram of a multimodal behavior detection method adapted to power scenarios, employing dual data augmentation and implicit inference. The architecture includes a data layer, a knowledge layer, a model layer, and an inference layer. The data layer utilizes a dual data augmentation strategy, starting with positive sample images of properly worn seatbelts and generating negative sample images of violations through two technical paths (controlled image editing and generative synthesis). The knowledge layer is used for multi-scenario, multi-rule semantic structuring and alignment, transforming complex power safety supervision rules into logic understandable to the model. The model layer is used for joint training of a large multimodal model, inputting the balanced multimodal dataset output from the data layer and the structured rules output from the knowledge layer into the large multimodal model. Through full-supervised fine-tuning, a fine-tuned detection model is obtained. The inference layer uses implicit rules to drive violation detection. Faced with an input image of a power operation to be inspected, it internally activates and matches implicit rules to determine violations related to seatbelt wearing.

[0074] In another embodiment, a second operation image is generated based on a first operation image using a preset data generation strategy, including: identifying a safety protection area in the first operation image through a preset semantic segmentation model; and removing protective equipment objects in the safety protection area through a preset image editing model under the constraints of maintaining the posture of the target object, the scene structure of the power operation scenario and the spatial relationship with the first operation image.

[0075] In practice, the terminal identifies the safety protection area in the first operation image using a preset semantic segmentation model. The semantic segmentation model refers to a deep learning model used to classify each pixel in the image. The preset semantic segmentation model employs a segmentation model based on a fully convolutional network, U-Net architecture, or Transformer architecture, and has been trained on a power operation scene image dataset.

[0076] The safety protection area refers to the pixel area occupied by the safety protection equipment worn by the target object in the first operation image, such as the strip-shaped area formed by the safety belt in the image, the hat-shaped area formed by the safety helmet in the image, or the hand area formed by the insulating gloves in the image.

[0077] In the specific implementation, the terminal inputs the first working image into the semantic segmentation model, and the semantic segmentation model outputs a segmentation mask with the same size as the first working image. Pixels belonging to the security protection area in the segmentation mask are labeled as foreground, and the remaining pixels are labeled as background. The terminal determines the position and boundary of the security protection area in the first working image based on the segmentation mask.

[0078] After obtaining the segmentation mask, the terminal performs a morphological dilation operation on the segmentation mask, expanding the boundary of the security protection area outward by a preset number of pixels. This ensures that the subsequent image editing model can cover the transition area between the protective equipment and the surrounding background when removing the protective equipment object, thus avoiding the retention of some pixels at the edge of the protective equipment after the removal operation.

[0079] Under the constraints of maintaining the posture of the target object, the spatial relationship between the scene structure of the power operation scenario and the first operation image, the terminal removes the protective equipment object in the safety protection area through a preset image editing model to obtain the second operation image.

[0080] The constraints include the unchanged limb posture of the target object, the unchanged relative positional relationship between the target object and other objects in the power operation scene, the unchanged background structure of the power operation scene, and the unchanged lighting conditions. The purpose of the constraints is to ensure that the second operation image generated after removing the protective equipment object is essentially consistent with the first operation image, with only the single variable of the presence or absence of safety protective equipment changing.

[0081] Among them, the image editing model refers to the deep learning model used to modify or fill the content of a specified area of ​​an image. The preset image editing model adopts an image inpainting model based on a diffusion model or an image inpainting model based on a generative adversarial network. After receiving the location information of the security protection area and the removal instruction, the image editing model fills the content of the area so that the filled area is consistent with the surrounding area in terms of texture and lighting.

[0082] For example, after removing the security protection area using an image editing model, the terminal can further smooth the boundaries of the edited area. This boundary smoothing refers to the terminal fusing pixels at the boundary between the security protection area and the surrounding background area. The fusing process employs feathering algorithms, multi-band fusion algorithms, or Laplacian pyramid fusion algorithms to ensure continuous texture transitions between the edited area and the surrounding areas, avoiding visual artifacts at the edited area boundaries. Through boundary smoothing, the terminal ensures that the second operational image maintains semantic consistency with the first operational image, except for the status of wearing security equipment, satisfying the single-variable control principle—that is, only the compliance status label is changed, without altering the essential content of the scene.

[0083] The technical solution of this embodiment accurately locates the security protection area through a semantic segmentation model, and removes the protective equipment object in a targeted manner through an image editing model while maintaining the constraint of keeping the scene structure unchanged. The generated second operation image retains the physical authenticity of the original scene and only changes the single variable of compliance status, which effectively alleviates the problem of model overfitting to background features. At the same time, the boundary smoothing process avoids the introduction of visual artifacts in the editing area, ensuring the image quality of the generated samples, so that the generated violation samples can truly reflect the visual characteristics of the violation behavior and improve the reliability of the sample dataset.

[0084] In another embodiment, a preset data generation strategy is used to generate a second operation image based on a first operation image, including: performing semantic parsing on the first operation image using a preset visual big data model to generate descriptive text for the first operation image; the descriptive text includes at least the environmental features, target object pose, and scene context information of the first operation image; and generating a second operation image excluding protective equipment objects based on the descriptive text using a preset text-to-image generation model.

[0085] In practice, the terminal uses a pre-set visual big data model to perform semantic parsing on the first task image, generating descriptive text for the first task image. The visual big data model refers to a deep learning model that integrates visual and linguistic information for joint modeling. The pre-set visual big data model employs a visual-language model based on a visual Transformer and a language model. This model has been pre-trained on large-scale image-text pairing data and possesses the ability to describe image content using natural language. Semantic parsing refers to the process by which the visual big data model understands the visual content in the first task image and outputs a structured or semi-structured text description.

[0086] The descriptive text includes at least the environmental features of the first work image, the target object's pose, and scene context information. Environmental features refer to the description of the work environment in the first work image, including scene type (e.g., power pole work scene, transmission tower work scene, or valve hall work scene), lighting conditions (e.g., daytime, nighttime, or cloudy), weather conditions (e.g., sunny, rainy, or foggy), and background objects (e.g., tower structure, power lines, fences, or ground material). Target object pose refers to the description of the target object's body pose in the first work image, including the target object's position (e.g., standing, crouching, hanging, or climbing), limb orientation (e.g., facing the camera, sideways to the camera, or away from the camera), and action type (e.g., climbing, maintenance, or carrying). Scene context information refers to the description of the relationship between the target object and surrounding objects in the first work image, including the relative position of the target object and power equipment, the relative position of the target object and protective facilities, and the stage of work the target object is in.

[0087] The terminal inputs the first task image into the visual model and constructs a prompt message to guide the model in generating descriptive text. This prompt message explicitly requires the descriptive text output by the visual model to cover three dimensions: environmental features, target object pose, and scene context information. It also requires the descriptive text to reflect that the target object is wearing safety protective equipment. Based on the input first task image and the prompt message, the visual model outputs the corresponding descriptive text.

[0088] The terminal uses a pre-defined text-to-image generation model to generate a second operational image excluding objects wearing protective equipment, based on descriptive text. The text-to-image generation model is a deep learning model that generates corresponding images based on input text descriptions. The pre-defined model employs either a diffusion-based generative model or a generative adversarial network-based model. This model has been pre-trained on large-scale image-text pairing data and possesses the ability to generate images based on text descriptions. The terminal inputs the descriptive text generated by the large-scale visual model into the text-to-image generation model and adds modifiers to the descriptive text to indicate that the target object in the generated image is not wearing protective equipment. The text-to-image generation model then generates the corresponding second operational image based on the modified descriptive text.

[0089] The technical solution of this application embodiment performs deep semantic analysis on the first operation image through a large visual model to generate descriptive text, and then generates violation sample images based on the descriptive text through a text-to-image generation model. This breaks the limitations of physical acquisition and can synthesize extreme violation scenarios that are extremely difficult to occur in reality, enriching the diversity and coverage of negative samples and effectively solving the long-tail distribution problem. At the same time, since the descriptive text includes environmental features, target object poses and scene context information, the generated second operation image is consistent with the real power operation environment in terms of scene content, ensuring the usability of the generated samples.

[0090] In another embodiment, constructing a sample dataset based on the first job image and the second job image includes: performing image enhancement operations on the first job image and the second job image respectively; constructing a sample dataset based on each first job image and each second job image after the image enhancement operations; and ensuring that the ratio between the first job image and the second job image in the sample dataset reaches a preset ratio.

[0091] Image enhancement operations refer to operations that expand the data scale and improve data diversity by performing geometric transformations or pixel value transformations on the original image. Image enhancement operations include one or more of the following operations: rotation, scaling, flipping, brightness perturbation, and noise injection.

[0092] Among them, rotation refers to the operation of rotating the image around the center of rotation by a preset angle with the center of rotation as the rotation center. The range of the preset angle is determined according to the posture stability of the target object in the image. The range of the preset angle is set to positive 5 degrees to positive 15 degrees and negative 5 degrees to negative 15 degrees to avoid the target object's posture being distorted due to the rotation angle exceeding the range.

[0093] Scaling refers to the operation of the terminal to enlarge or reduce the image according to a preset scaling factor. The value range of the scaling factor is determined according to the proportion of the target object in the image. The value range of the scaling factor is set to 0.8 to 1.2 to avoid the target object from exceeding the reasonable range in the image after scaling.

[0094] Among them, flipping refers to the operation of the terminal to mirror and flip the image in the horizontal or vertical direction. Horizontal flipping is suitable for operation scenarios with symmetrical backgrounds, while vertical flipping is suitable for scenarios where the pose of the target object allows it to be upside down.

[0095] Among them, brightness perturbation refers to the operation of the terminal to adjust the pixel brightness value of the image as a whole. The adjustment range of brightness perturbation is determined according to the light adaptation capability of the image acquisition device. The value range of the brightness adjustment coefficient is set from 0.7 to 1.3 to simulate the working scenarios under different lighting conditions.

[0096] Noise injection refers to the operation of the terminal superimposing random noise into the image. The noise type includes Gaussian noise, salt-and-pepper noise, or Poisson noise. The noise intensity is determined according to the noise characteristics of the image sensor to simulate the noise generated by the image acquisition device in a real environment.

[0097] The terminal constructs a sample dataset based on the first and second job images after image enhancement operations. During image enhancement, the terminal applies the same combination of enhancement parameters to both the first and second job images to ensure consistent data distribution between the enhanced positive and negative samples. The terminal uses the enhanced first job images as positive samples and the enhanced second job images as negative samples, constructing the sample dataset according to a preset data organization format. The ratio of the number of first and second job images in the sample dataset reaches a preset ratio.

[0098] The preset ratio refers to the ratio of positive to negative samples in the sample dataset, set to 1:1, meaning the number of positive and negative samples are equal. Since there are sufficient compliant samples and scarce non-compliant samples in real power operation scenarios, the terminal uses a data generation strategy to generate a second operation image and an image enhancement operation to expand the number of negative samples to be equal to the number of positive samples, achieving a balanced state of positive and negative samples in the sample dataset. When constructing the sample dataset, the terminal labels each sample with the corresponding scenario type, operation height, protection conditions, operation location, and compliance or non-compliance label, and divides the labeled samples into training, validation, and test sets.

[0099] In this embodiment, by performing image enhancement operations, including rotation, scaling, flipping, brightness perturbation, and noise injection, on the first and second task images, the scale of the sample dataset is expanded and the diversity of samples under geometric transformations and pixel value transformations is improved, enabling the model to learn feature representations under different transformation conditions during training. At the same time, by setting the ratio of positive to negative samples to a preset ratio, a balance between positive and negative samples is achieved, fundamentally alleviating the model's prediction bias towards positive samples and improving the model's sensitivity and accuracy in identifying violations.

[0100] In another embodiment, training the multimodal large model to be trained based on the sample dataset and structured rule knowledge to obtain the trained multimodal large model includes: converting the structured rule knowledge into structured prompt text; taking the sample dataset and the corresponding structured prompt text as input and the corresponding violation judgment result as the expected output, fine-tuning the multimodal large model to be trained so that the multimodal large model to be trained establishes an alignment relationship between visual features, scene semantics and rule logic, thereby obtaining the trained multimodal large model.

[0101] Among them, structured cue text refers to cue information that expresses structured rule knowledge in natural language form and is used as text input to guide the model to learn rule logic during the model training phase. The cue information is the input text used to guide the model to understand the task or inject prior knowledge.

[0102] In specific implementation, the terminal converts the mapping relationship between the work scenario characteristics, safety operation rules, and violation judgment results in the structured rule knowledge into structured prompt text according to a preset text template. The text template format is as follows: the characteristics of the current work scenario are: scenario type, work height is: height value, protection conditions are: protection conditions, and work location is: work location; according to the safety operation rules, the content of the safety operation rules; if the target object does not meet the above requirements, the judgment result is a violation; otherwise, the judgment result is compliance. For each sample in the sample dataset, the terminal generates the corresponding structured prompt text based on the scene type, work height, protection conditions, work location, and safety operation rules corresponding to that sample.

[0103] The terminal takes the sample dataset and the corresponding structured prompt text as input, and the corresponding violation judgment result as the expected output. It performs model fine-tuning on the multimodal large model to be trained, so that the multimodal large model to be trained establishes the alignment relationship between visual features, scene semantics and rule logic, and obtains the trained multimodal large model.

[0104] Model fine-tuning refers to the training process of further updating the model parameters using sample datasets and structured rule knowledge based on the existing pre-trained parameters of the multimodal large model to be trained. Model fine-tuning can be done in two ways: full fine-tuning, which updates all model parameters; or parameter-efficient fine-tuning, such as introducing a low-rank matrix into the model's attention layer using a low-rank adaptation method and updating only the parameters of the low-rank matrix, in order to reduce the computational resource consumption of the training process.

[0105] During model fine-tuning, the terminal inputs the task images from the sample dataset into the visual encoder of the multimodal large-scale model to be trained. The visual encoder extracts the visual features of the task images. The corresponding structured prompt text is input into the language decoder of the multimodal large-scale model to be trained. The language decoder encodes the structured prompt text to obtain textual representations of scene semantics and rule logic. The multimodal large-scale model to be trained interacts with the visual features and textual representations through a cross-modal attention mechanism, establishing an alignment relationship between visual features, scene semantics, and rule logic, and outputs the corresponding violation judgment result. The terminal calculates the cross-entropy loss between the violation judgment result output by the model and the expected output, and updates the model parameters through backpropagation until the loss function converges, thus obtaining the trained multimodal large-scale model.

[0106] The alignment relationship refers to the mapping relationship between the visual feature space, scene semantic space and rule logic space established during the training process of the trained multimodal large model. The alignment relationship enables the model to identify the corresponding work scene features based on the visual features of the input image, activate the corresponding safety operation rules based on the work scene features, and then infer the violation judgment result based on the safety operation rules.

[0107] In this embodiment, by transforming structured rule knowledge into structured prompt text and using the sample dataset and structured prompt text together as input for model fine-tuning, the multimodal large model to be trained not only learns the visual features of the task images during training, but also learns the correspondence between scene semantics and rule logic. This establishes an alignment relationship between visual features, scene semantics, and rule logic, enabling the model to possess the joint ability of rule understanding and scene recognition. This provides a foundation for the model to adaptively call the corresponding rules based on scene features during subsequent inference. At the same time, by using the violation judgment result as the expected output for supervised training, the model's output results have clear judgment criteria, improving the interpretability of the model's judgment results.

[0108] In another embodiment, after training the multimodal large model to be trained based on the sample dataset and structured rule knowledge to obtain the trained multimodal large model, the method further includes: inputting the job image to be detected into the trained multimodal large model; extracting the visual features and scene semantics of the job image to be detected through the trained multimodal large model, activating safety operation rules that match the job scene features of the job image to be detected based on alignment relationships, and outputting the behavior detection result for whether the job image to be detected belongs to a violation of operation behavior.

[0109] Among them, the operation image to be detected refers to the operation image that needs to be detected in real time by the terminal through the image acquisition equipment deployed in the power operation scenario. The compliance status of the target object in the operation image to be detected is unknown and needs to be determined by a trained multimodal large model.

[0110] In practice, the terminal extracts the visual features and scene semantics of the work image to be detected through the trained multimodal large model, and activates the safety operation rules that match the work scene features of the work image to be detected based on the alignment relationship, and outputs the behavior detection result of whether the work image to be detected belongs to the violation of the operation behavior.

[0111] Visual features refer to the feature vectors representing the visual content in the image after the visual encoder of the trained multimodal large model extracts features from the image to be detected. Visual features include the pose features of the target object, the presence or absence of safety protection equipment, and scene background features. Scene semantics refer to the semantic information representing the attributes of the work scene corresponding to the image to be detected, inferred by the trained multimodal large model based on the visual features. Scene semantics includes dimensions such as scene type, work height, protection conditions, and work location.

[0112] For example, the process by which the terminal activates safety operation rules that match the job scene features of the job image to be detected based on the alignment relationship using a trained multimodal big model is as follows: The trained multimodal big model retrieves job scene feature nodes that match the current job scene features in the mapping space defined by the alignment relationship according to the extracted visual features and scene semantics, and then activates the safety operation rule corresponding to the job scene feature node according to the mapping relationship in the structured rule knowledge. This process does not require the terminal to explicitly input structured rule knowledge. The trained multimodal big model implicitly completes the rule activation through the learned alignment relationship, realizing the transformation from explicit rule dependence to implicit rule-driven capability.

[0113] The terminal, through a trained multimodal large model, infers behavior detection results based on activated safety operation rules and the protective equipment wearing status of the target object in the work image to be detected. The behavior detection results include two judgment conclusions: violation and compliance. When the judgment conclusion is violation, the behavior detection results also include the corresponding violation type (e.g., not wearing a safety belt, not wearing a safety helmet, or not wearing insulated protective equipment), the content of the activated safety operation rule, and the basis for the judgment; when the judgment conclusion is compliance, the behavior detection results also include corresponding compliance confirmation information. The terminal outputs the behavior detection results to a display device for presentation or to an alarm device to trigger corresponding alarm operations.

[0114] In this embodiment, the image of the operation to be detected is input into a trained multimodal large model. The model extracts visual features and scene semantics and implicitly activates the matching safety operation rules based on alignment relationships, outputting the behavior detection results. This allows the model to adaptively call the corresponding rules to complete the violation judgment based on the current scene features without explicitly inputting rule information during the inference stage. This realizes implicit rule-driven inference, improves the model's generalization ability under multiple scene switching or rule change conditions, and enables the model to adapt to diverse power operation environments, meeting the reliability requirements of actual power safety monitoring systems for behavior detection in complex scenarios.

[0115] For the convenience of those skilled in the art, Figure 4 An illustrative diagram illustrating the input and output of a multimodal large-scale model is provided. During the actual inference phase, a real-world image of a power operation to be detected (e.g., a real-time snapshot of personnel on a high tower, slope, or mobile platform) can be input, eliminating the need for explicit textual safety rule descriptions. Upon receiving the input image, the trained multimodal large-scale model's internal network automatically extracts the image's visual features and complex scene semantics. Based on the alignment relationships established during training, it adaptively and implicitly activates safety rule judgment logic matching the current scene (e.g., identifying a tower environment greater than 2 meters high). The model can directly output the final behavior detection result. This output accurately indicates a binary classification state of "violation" or "compliance" (e.g., {"Is the worker safe?": "Safe"}), and also outputs the scene features relied upon by the model during implicit inference, as well as the specific safety procedure entries activated, improving the generalization ability and interpretability of the power safety monitoring system in complex and variable operating environments.

[0116] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0117] Based on the same inventive concept, this application also provides a device for implementing the multimodal behavior detection method for adapting to power scenarios using dual data augmentation and implicit reasoning, as described above. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations of one or more embodiments of the multimodal behavior detection device for adapting to power scenarios using dual data augmentation and implicit reasoning provided below can be found in the limitations of the multimodal behavior detection method for adapting to power scenarios using dual data augmentation and implicit reasoning described above, and will not be repeated here.

[0118] In one exemplary embodiment, such as Figure 5 As shown, a multimodal behavior detection device with dual data augmentation and implicit reasoning adapted to power scenarios is provided, including:

[0119] The acquisition module 510 is used to acquire a first operation image of a power operation scenario, wherein the first operation image includes an image of a target object performing compliant operation behavior;

[0120] The construction module 520 is used to generate a second job image based on the first job image using a preset data generation strategy, and to construct a sample dataset based on the first job image and the second job image; the data generation strategy includes local modification based on image content and image generation based on semantic description; the second job image includes an image of a target object performing an illegal operation.

[0121] The generation module 530 is used to generate structured rule knowledge for the power operation scenario; the structured rule knowledge includes the mapping relationship between operation scenario characteristics, safe operation rules, and violation judgment results;

[0122] The training module 540 is used to train the multimodal large model to be trained based on the sample dataset and the structured rule knowledge, so as to obtain the trained multimodal large model; the multimodal large model is used to output the corresponding behavior detection result based on the task image to be detected.

[0123] In one embodiment, the construction module 520 is specifically used to identify the safety protection area in the first operation image through a preset semantic segmentation model; under the constraint of maintaining the posture of the target object and the spatial relationship between the scene structure of the power operation scene and the first operation image, the protective equipment object in the safety protection area is removed through a preset image editing model to obtain the second operation image.

[0124] In one embodiment, the construction module 520 is specifically used to perform semantic parsing on the first operation image through a preset visual large model to generate descriptive text for the first operation image; the descriptive text includes at least the environmental features, target object pose, and scene context information of the first operation image; and generates a second operation image excluding protective equipment objects based on the descriptive text through a preset text-to-image generation model.

[0125] In one embodiment, the construction module 520 is specifically used to perform image enhancement operations on the first job image and the second job image respectively; the image enhancement operation includes one or more operations such as rotation, scaling, flipping, brightness perturbation and noise injection; based on each of the first job images and each of the second job images after the image enhancement operation, the sample dataset is constructed; the ratio between the number of the first job images and the number of the second job images in the sample dataset reaches a preset ratio.

[0126] In one embodiment, the training module 540 is specifically used to convert the structured rule knowledge into structured prompt text; take the sample dataset and the corresponding structured prompt text as input, and the corresponding violation judgment result as the expected output to fine-tune the multimodal large model to be trained, so that the multimodal large model to be trained establishes an alignment relationship between visual features, scene semantics and rule logic, and obtain the trained multimodal large model.

[0127] In one embodiment, the multimodal behavior detection device further includes a detection module, which is specifically used to input the work image to be detected into the trained multimodal large model; extract the visual features and scene semantics of the work image to be detected through the trained multimodal large model; activate safety operation rules that match the work scene features of the work image to be detected based on the alignment relationship; and output the behavior detection result for whether the work image to be detected belongs to a violation of the operation behavior.

[0128] The modules in the aforementioned multimodal behavior detection device with dual data augmentation and implicit inference adapted to power scenarios can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0129] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a multimodal behavior detection method that adapts to power scenarios with dual data augmentation and implicit inference. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0130] Those skilled in the art will understand that Figure 6The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0131] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0132] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0133] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0134] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0135] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0136] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0137] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multimodal behavior detection method adapted to power scenarios using dual data augmentation and implicit reasoning, characterized in that, The method includes: Acquire a first operational image of a power operation scenario, wherein the first operational image includes an image of the target object performing compliant operational behavior; A second task image is generated based on the first task image using a preset data generation strategy, and a sample dataset is constructed based on the first task image and the second task image; the data generation strategy includes local modification based on image content and image generation based on semantic description; the second task image includes an image of a target object performing an illegal operation. Generate structured rule knowledge for the power operation scenario; the structured rule knowledge includes the mapping relationship between operation scenario characteristics, safe operation rules, and violation judgment results; The multimodal large model to be trained is trained based on the sample dataset and the structured rule knowledge to obtain the trained multimodal large model; the multimodal large model is used to output the corresponding behavior detection results based on the job image to be detected.

2. The method according to claim 1, characterized in that, The step of generating a second job image based on the first job image using a preset data generation strategy includes: The safety protection area in the first operation image is identified by a preset semantic segmentation model; Under the constraints of maintaining the posture of the target object and the spatial relationship between the scene structure of the power operation scenario and the first operation image, the protective equipment object in the safety protection area is removed by a preset image editing model to obtain the second operation image.

3. The method according to claim 1, characterized in that, The step of generating a second job image based on the first job image using a preset data generation strategy includes: The first task image is semantically parsed using a pre-defined visual model to generate descriptive text for the first task image; the descriptive text includes at least the environmental features, target object pose, and scene context information of the first task image. A second operational image, excluding protective equipment objects, is generated based on the descriptive text using a preset text-to-image generation model.

4. The method according to any one of claims 1 to 3, characterized in that, The step of constructing a sample dataset based on the first job image and the second job image includes: Image enhancement operations are performed on the first job image and the second job image respectively; the image enhancement operations include one or more of the following: rotation, scaling, flipping, brightness perturbation, and noise injection. Based on each of the first job images and each of the second job images after the image enhancement operation, the sample dataset is constructed; the ratio of the number of the first job images and the number of the second job images in the sample dataset reaches a preset ratio.

5. The method according to claim 1, characterized in that, The step of training the multimodal large model to be trained based on the sample dataset and the structured rule knowledge to obtain the trained multimodal large model includes: The structured rule knowledge is transformed into structured prompt text; The sample dataset and the corresponding structured prompt text are used as input, and the corresponding violation judgment result is used as the expected output. The multimodal large model to be trained is fine-tuned so that the multimodal large model to be trained establishes an alignment relationship between visual features, scene semantics and rule logic, and the trained multimodal large model is obtained.

6. The method according to claim 5, characterized in that, After training the multimodal large model to be trained based on the sample dataset and the structured rule knowledge to obtain the trained multimodal large model, the method further includes: The image of the task to be detected is input into the trained multimodal large model; The trained multimodal large model extracts the visual features and scene semantics of the work image to be detected, and activates safety operation rules that match the work scene features of the work image to be detected based on the alignment relationship, and outputs the behavior detection result of whether the work image to be detected belongs to a violation of the operation behavior.

7. A multimodal behavior detection device adapted to power scenarios with dual data augmentation and implicit reasoning, characterized in that, The device includes: The acquisition module is used to acquire a first operation image of a power operation scenario, wherein the first operation image includes an image of a target object performing compliant operation behavior; A construction module is used to generate a second task image based on the first task image using a preset data generation strategy, and to construct a sample dataset based on the first task image and the second task image; the data generation strategy includes local modification based on image content and image generation based on semantic description; the second task image includes an image of a target object performing an illegal operation. The generation module is used to generate structured rule knowledge for the power operation scenario; the structured rule knowledge includes the mapping relationship between operation scenario characteristics, safe operation rules, and violation judgment results; The training module is used to train the multimodal large model to be trained based on the sample dataset and the structured rule knowledge, so as to obtain the trained multimodal large model; the multimodal large model is used to output the corresponding behavior detection results based on the task image to be detected.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images