Industrial unsafe behavior detection method, system and device and storage medium
By fine-tuning the large visual language model through reinforcement learning and designing a structured text format, the problems of cross-scene adaptability and multi-instance detection in industrial unsafe behavior detection are solved, achieving high-precision and interpretable behavior detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-26
- Publication Date
- 2026-03-27
AI Technical Summary
Existing methods for detecting unsafe behaviors in industry suffer from limitations such as reliance on high-quality labeled data, insufficient cross-scenario adaptability, and limited semantic understanding and logical reasoning capabilities, making it difficult to accurately identify complex behavioral patterns. Furthermore, the lack of multi-instance detection capabilities in large-scale visual-language models leads to insufficient detection accuracy and interpretability.
We employ reinforcement learning algorithms to fine-tune a large visual language model, combine it with a structured multimodal question-and-answer text format, and optimize bounding box prediction and language generation through reward signals. This guides the model to output structured multimodal question-and-answer text, enabling cross-modal understanding and multi-object detection.
It improves the location accuracy and cross-modal reasoning ability of industrial unsafe behavior detection, and can understand natural language safety rules and generate interpretable thought chains, significantly improving detection accuracy and system reliability.
Smart Images
Figure CN121747154A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of industrial unsafe behavior detection technology, and in particular to a method, system, device and storage medium for industrial unsafe behavior detection. Background Technology
[0002] In industrial production safety scenarios, a single image often contains multiple individuals engaging in unsafe behaviors, requiring multi-instance detection capabilities for detection. However, in personnel behavior detection tasks involving multimodal safety rule reasoning, there is a common problem of weak reasoning ability, particularly in cross-modal understanding (such as visual-text joint modeling) and semantic-guided analysis.
[0003] Currently, research on industrial safety detection algorithms largely focuses on traditional small-scale deep learning models, such as Transformer-based detectors (e.g., Co-DETR proposed by Zong et al. in the paper "Detrs with collaborative hybrid assignments training. Proceedings of the IEEE / CVF international conference on computer vision. 2023") and the YOLO series widely used in industrial scenarios (e.g., YOLOv8 proposed by Muhammad in the paper "What is YOLOv8: An In-Depth Exploration of the Internal Features of the Next-Generation Object Detector. arXiv preprint arXiv:2408.15857. 2024"). These methods are mainly aimed at object detection or classification tasks, relying on supervised learning paradigms and training on a large number of labeled samples on specific datasets to identify and locate unsafe behaviors or abnormal operations. Although they exhibit good detection performance in closed environments, they face multiple challenges in practical applications.
[0004] First, the high dependence of these methods on high-quality, finely labeled data limits their widespread application. Because abnormal behavior samples in industrial environments are scarce and diverse, and require domain experts for annotation, the acquisition cost is high and the total amount is limited, affecting the quality of model training. Second, existing models generally lack cross-scenario adaptability. Once the production environment changes, the performance of the original model significantly declines, necessitating the collection of labeled data for the new scenario and retraining or fine-tuning, increasing deployment and maintenance costs and failing to meet the needs of intelligent manufacturing systems for rapid iteration and flexible adaptation. Furthermore, traditional small models have limited capabilities in semantic understanding and contextual reasoning. Relying solely on visual feature matching, they struggle to accurately judge complex behavioral patterns and lack the ability to understand textual rules such as safety operating procedures. They cannot logically associate complex behaviors such as "not wearing a safety belt correctly," "ladder unsupported," or "work at height without supervision" with regulatory instructions, nor can they guide the model to perform rule-based and interpretable analysis of image content through natural language prompts (e.g., "Please check if anyone has violated safety regulations"). This limitation severely restricts the system's ability to identify complex unsafe behaviors when facing rapidly changing industrial scenarios, necessitating the introduction of stronger semantic understanding and logical reasoning capabilities.
[0005] In recent years, with the development of Vision-Language Models (VLMs), such as LLaVA proposed by Liu et al. in "Visual instruction tuning. Advances in neural information processing systems. 2023" and Qwen2.5-VL proposed by Bai et al. in "Qwen2.5-vl technical report. arXivpreprint arXiv:2502.13923. 2025", these models have demonstrated good generalization and cross-modal alignment capabilities, enabling them to understand textual semantics and respond to natural language instructions. However, many general-purpose VLMs still face a key challenge in the vertical field of industrial safety: a lack of conversational multi-instance detection capabilities. Specifically, when a user issues the instruction "Please indicate all violations in the image and their locations," existing VLMs often only generate general text descriptions (such as "Someone is not wearing a safety helmet"), and cannot simultaneously output multiple precise bounding boxes to locate different unsafe behaviors (such as not wearing a seatbelt).
[0006] A deeper problem lies in the training paradigm. Some approaches attempt to fine-tune the VLM through pure text supervision, directly treating the bounding box coordinates of object detection (e.g., [x1, y1, x2, y2]) as strings (e.g., "[100,150,130,200]") and using them as the output sequence of the language model, optimizing with standard language modeling losses (e.g., cross-entropy). However, this approach ignores the object detection characteristics of the bounding boxes in spatial geometry. As a result, although the text description seems reasonable, the generated bounding boxes often exhibit offsets, distortions, or missed detections, severely impacting the localization accuracy and system reliability in actual deployments. Unsafe behavior detection in industrial production safety scenarios requires not only detection capabilities but also the ability to reason according to safety regulations, necessitating solutions to bottlenecks in semantic-geometric joint modeling, multi-object detection, and reasoning capabilities.
[0007] In view of this, the present invention is hereby proposed. Summary of the Invention
[0008] The purpose of this invention is to provide a method, system, device, and storage medium for detecting unsafe industrial behaviors, which can accurately and across modally detect unsafe industrial behaviors.
[0009] The objective of this invention is achieved through the following technical solution: A method for detecting unsafe industrial behaviors includes: Annotated image data of industrial production safety scenarios are collected, and text instructions in a structured multimodal question-and-answer text format are combined with reinforcement learning algorithms to fine-tune a large visual language model. The process includes: inputting image data and corresponding text instructions into the large visual language model to obtain the response text of the large visual language model; calculating the reward based on the format of the response text and the position of the predicted bounding box and the labeled true bounding box in the response text; and combining the reward calculation to optimize the objective function and fine-tune the large visual language model. After fine-tuning, the text instructions and the image to be detected are input into the fine-tuned visual language model, which then guides the visual language model to output a structured multimodal question-and-answer text format response.
[0010] An industrial unsafe behavior detection system for implementing the aforementioned method includes: The model fine-tuning unit is used to collect labeled image data of industrial production safety scenarios and combine it with text instructions in a structured multimodal question-and-answer text format. It uses reinforcement learning algorithms to fine-tune the visual language model, including: inputting image data and corresponding text instructions into the visual language model, obtaining the response text of the visual language model, calculating the reward based on the format of the response text and the position of the predicted bounding box and the labeled true bounding box in the response text, and combining the reward calculation to optimize the objective function to fine-tune the visual language model. The industrial unsafe behavior detection unit, after fine-tuning, inputs text commands and the image to be detected into the fine-tuned visual language model, guiding the visual language model to output a structured multimodal question-and-answer text format response.
[0011] A processing device includes: one or more processors; and a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.
[0012] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.
[0013] As can be seen from the technical solutions provided by the present invention above, (1) the detection of industrial unsafe behaviors based on the visual language large model can simultaneously understand and analyze natural language safety rules and detect related targets; (2) GRPO and other RL (Reinforcement Learning) algorithms are applied to the task of industrial unsafe behavior detection, and the reasoning ability and detection accuracy are significantly improved by optimizing language generation and bounding box prediction through reward signals; thanks to the above improvements, the present invention can solve the problems of insufficient positioning accuracy and cross-modal reasoning ability in industrial unsafe behavior detection. Attached Figure Description
[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0015] Figure 1 This is a flowchart of an industrial unsafe behavior detection method provided in an embodiment of the present invention.
[0016] Figure 2 This is a schematic diagram of the model training scheme provided in an embodiment of the present invention.
[0017] Figure 3 This is a schematic diagram of the model reasoning process provided in an embodiment of the present invention.
[0018] Figure 4 This is a visual diagram of the input and output of the Qwen2.5-VL-7B provided in an embodiment of the present invention.
[0019] Figure 5 This is a schematic diagram of the input and output of a model trained by reinforcement learning, provided in an embodiment of the present invention.
[0020] Figure 6 This is a schematic diagram of an industrial unsafe behavior detection system provided in an embodiment of the present invention.
[0021] Figure 7 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation
[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.
[0023] First, the following explanations are provided for the terms that may be used in this article: The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.
[0024] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.
[0025] The following is a detailed description of a method, system, device, and storage medium for detecting unsafe industrial behavior provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they shall be performed according to conventional conditions in the art or conditions recommended by the manufacturer. Where the manufacturers of reagents or instruments used in the embodiments of this invention are not specified, they are all conventional products that can be purchased commercially.
[0026] Example 1 This invention provides a method for detecting unsafe industrial behaviors, such as... Figure 1 As shown, it mainly includes the following steps: Step 1: Model fine-tuning stage based on reinforcement learning algorithm.
[0027] Annotated image data of industrial production safety scenarios are collected, and text instructions in a structured multimodal question-and-answer text format are combined with reinforcement learning algorithms to fine-tune a large visual language model. The process includes: inputting image data and corresponding text instructions into the large visual language model to obtain the response text of the large visual language model; calculating the reward based on the format of the response text and the position of the predicted bounding box and the labeled true bounding box in the response text; and combining the reward calculation to optimize the objective function (reward expectation) to fine-tune the large visual language model.
[0028] In this embodiment of the invention, the text instructions in the structured multimodal question-and-answer text format include: placeholders for image features, used to insert image features extracted by the visual language big model; security specification description text; and instruction text to guide the visual language big model in reasoning.
[0029] In this embodiment of the invention, the model response section is defined, which mainly includes three parts: reasoning process, reasoning conclusion, and predicted bounding box answer, and each part has corresponding labels at the beginning and end.
[0030] In this embodiment of the invention, the calculation of the reward based on the format of the answer text and the position of the predicted bounding box and the labeled true bounding box in the answer text includes: calculating the format reward, which includes: label format reward, bounding box format reward, and non-repetition format reward. The three rewards are combined as the format reward to constrain the format of the answer text of the visual language model and guide the visual language model to generate a structured multimodal question-and-answer text format answer; and the accuracy reward, which includes: calculating various different metrics using the position of the bounding box and the true bounding box, and determining the accuracy reward based on the corresponding metric values.
[0031] In this embodiment of the invention, the label format reward includes: if the visual language model's response text outputs the reasoning process, reasoning conclusion, and predicted bounding box answer according to the label format set in the model response section, the reward value is 1; otherwise, it is 0. The bounding box format reward includes: if the format of the predicted bounding box answer in the visual language model's response text is the format set in the model response section, the reward value is 1; otherwise, it is 0. The non-repeating format reward includes: segmenting the reasoning process in the visual language model's response text by sentence and detecting whether there is repetition; if there are no repetitive sentences, the reward value is 1; otherwise, it is 0.
[0032] In this embodiment of the invention, the accuracy reward includes: multi-objective IoU reward and L1 reward; wherein, the multi-objective IoU reward uses the IoU between the predicted bounding box and the ground truth bounding box as a metric, and the multi-objective IoU reward is determined based on the metric value; the L1 reward uses the L1 distance between the center point of the bounding box and the center point of the ground truth bounding box as a metric, and the L1 reward is determined based on the metric value.
[0033] In this embodiment of the invention, the predicted bounding box answer is denoted as ;in, Let M be the i-th predicted bounding box, and M be the total number of predicted bounding boxes; let the ground truth bounding box be denoted as . ,in, For the first There are N true bounding boxes; the Hungarian algorithm is used to find the optimal one-to-one assignment, assigning a unique match between each predicted bounding box and a true bounding box; during multi-objective IoU reward calculation, for any matched predicted bounding box and true bounding box, if the IoU exceeds a set first threshold, the reward is increased. The `max` function outputs the maximum value within the parentheses. During L1 reward calculation, for any matching predicted bounding box and ground truth bounding box, if the L1 distance is less than a set second threshold, the reward is increased. .
[0034] In this embodiment of the invention, the objective function optimized by combining reward calculation includes:
[0035] Let the strategy of the large visual language model before the fine-tuning in the last training round be denoted as... The model outputs the answer text, denoted as The corresponding reward is recorded as ,in, For the k-th answer text, The reward for the k-th response text, where K is the number of response texts, is calculated using the following formula: ; in, It represents the relative advantage of the k-th response text; mean(.) is the mean function, and std(.) is the standard deviation function; Let G be the policy of the large visual language model after the fine-tuning in the previous training round, and calculate the optimization objective function using the following formula: ; Where E represents the expectation, and This refers to strategy G and Use input x to perform reasoning and output the answer text. The conditional probability distribution; input x refers to text instructions and image data; clip is the truncation operation. The truncation range parameter will be used for the truncation operation. Limited to Within the specified range, prevent excessively large updates; The multiplication coefficients of the KL divergence control the degree of conservatism in the update. As a reference strategy, it is the conditional probability distribution of the frozen initial model; It is policy G and reference policy The KL divergence between strategies measures the difference between them.
[0036] Step 2: Inference results based on the fine-tuned model.
[0037] In this embodiment of the invention, after fine-tuning, the text command and the image to be detected are input into the fine-tuned visual language model, guiding the visual language model to output a structured multimodal question-and-answer text format response text.
[0038] Based on the above introduction, the structured multimodal question-and-answer text format mainly includes: the reasoning process, the reasoning conclusion, and the predicted bounding box answer.
[0039] The solution provided in this invention significantly improves the visual language large model's cross-modal understanding, fine-grained localization, and security rule analysis capabilities under complex natural language instructions by using a visual language large model (VLM) architecture, a reinforcement learning-driven inference optimization mechanism, and a structured output design. This visual language large model can not only accurately parse security specifications expressed by users in natural language, but also generate interpretable thought chain reasoning processes and simultaneously output high-precision target bounding boxes, effectively overcoming the limitations of traditional methods in inference capabilities and multimodal alignment. The main advantages of this invention are as follows: (1) A method for detecting unsafe industrial behaviors based on a large visual language model is proposed, which can simultaneously understand and analyze natural language safety rules and detect related targets.
[0040] (2) Applying RL algorithms such as GRPO to industrial unsafe behavior detection tasks, by jointly optimizing language generation and bounding box prediction through reward signals, and introducing special labels to explicitly model intermediate inference steps, significantly improves inference ability and detection accuracy.
[0041] (3) Embed safety production standards as prior knowledge into the prompt design to guide the visual language big model to make judgments and conduct detection according to the rules.
[0042] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.
[0043] I. Overall Introduction to the Plan.
[0044] To address the shortcomings of existing methods in terms of positioning accuracy and cross-modal reasoning capabilities, this invention proposes an industrial unsafe behavior detection scheme with strong cross-modal reasoning capabilities. This scheme can not only understand safety rules but also perform positioning and behavior judgment in natural language interaction.
[0045] First, a large-scale visual language model with visual thinking and reasoning capabilities is introduced to process text and visual features. By aligning image feature encoding with the language space, it can perform reasoning based on implicit human instructions, thus overcoming the limitations of traditional methods in reasoning ability and cross-modal interaction. This allows the large-scale visual language model to accurately analyze and reason about the test image according to user-defined safety rules. While generating natural language responses, the large-scale visual language model can predict the spatial location of the corresponding target, achieving integration of language output and detection boxes.
[0046] This invention also introduces reinforcement learning (RL) to effectively enhance the reasoning ability of the Visual Language Model (VLM) in visual perception tasks. Compared with traditional methods, the RL-optimized model also exhibits stronger generalization performance in unseen scenes. The introduction of the chain-of-thought (CoT) mechanism further enhances its generalization performance, enabling the large visual language model to decompose complex instructions into a series of ordered analysis steps, thereby effectively processing complex instructions and achieving accurate localization of target objects. In contrast, Supervised Fine-Tuning (SFT), which relies solely on text for training, typically lacks modeling of intermediate reasoning processes, thus limiting its generalization ability in unseen scenes. By employing reinforcement learning algorithms such as Group Relative Policy Optimization (GRPO), the large visual language model is guided to generate structured reasoning paths under the drive of reward signals, activating its inherent cognitive reasoning mechanism. Simultaneously, the reward signals contain constraints on the generated bounding boxes, significantly improving the accuracy of bounding box prediction. RL-driven training paradigms can effectively induce models to form interpretable and traceable thought chains, thereby achieving more robust and accurate performance in complex visual localization tasks.
[0047] To further enhance the model's ability to understand and apply safety regulations, this invention designs a structured question-and-answer template and introduces special tags. <think>< / think> , <conclusion>< / conclusion> , <answer>< / answer> This allows the model to output structured information such as coordinates required for the task, facilitating subsequent processing and making the coordinates output by the visual language model in text responses more accurate. Furthermore, it incorporates safety production rule hints to guide the model's reasoning and judgment, endowing it with the ability to detect specific targets. This not only improves the accuracy of the visual language model in understanding natural language instructions but also supports simultaneous detection and localization of multiple instances.
[0048] Ultimately, the visual language big model not only supports interaction with natural language prompts, but also has the ability to perceive, identify, and interpret various unsafe behaviors in complex industrial scenarios. This enables the discovery of potential safety hazards, significantly improving the interpretability, flexibility, and practical value of industrial unsafe behavior detection methods, and effectively enhancing the overall level of safe production.
[0049] II. Detailed introduction of the plan.
[0050] The following sections will introduce the data format, training scheme, and inference process respectively.
[0051] 1. Data format.
[0052] To achieve fine-grained understanding and anomaly identification of personnel behavior in industrial production scenarios, a batch of image data from industrial production safety scenarios were labeled, and the large visual language model was fine-tuned in this scenario.
[0053] Simultaneously, a structured multimodal question-answering text format was designed to guide the model in generating responses with semantic descriptions and spatial positioning information based on its understanding of image content. This format not only supports the recognition of target objects in images but also significantly enhances the ability to detect specified targets at the region level by introducing bounding box coordinates. For unsafe behavior detection tasks in specific industrial production scenarios, the visual-language large model can not only understand the entire scene but also locate and describe the region of unsafe behavior, achieving intuitive object association. It realizes functions such as region understanding, object localization, and dialogue interaction, making vision-language-based task processing more intelligent and efficient.
[0054] The multimodal question-answering text format mainly includes: placeholders for image features, used to insert image features extracted by the visual language big model; security specification description text; and instruction text to guide the visual language big model in reasoning.
[0055] For example, the following template can be used: USER: When working at heights, safety belts, safety helmets, and complete work clothes must be worn correctly, and insulated gloves should be worn as needed. Safety belts must be secured to a sturdy object to prevent them from slipping. Safety belts should be attached high and used low. When using a single ladder, the ladder should be supported by someone. A dedicated person should be stationed below the work site to monitor the area. Please describe whether the workers in the image meet the above requirements, output the coordinates of the workers in the image in the following format, and summarize the unsafe behaviors of the workers in the image.
[0056] ASSISTANT: <think> The worker in the picture is operating a power transformer at a height. He is wearing a safety helmet, which meets safety requirements, but no safety belt or gloves are visible. He is standing on a ladder, and there is no indication that another person is holding the ladder securely. No dedicated safety supervisor is also visible in the image. Furthermore, the worker is not wearing full protective clothing, which is required by safety regulations.< / think> <conclusion> The workers wore safety helmets and work clothes correctly, but did not wear safety belts and insulated gloves. The safety belts were not secured to a sturdy object, the ladder was unsupported, and there was no one supervising the work area below.< / conclusion> <answer> [{"bbox": [245, 223, 390, 679]}]< / answer> in: This is a placeholder for image features, representing the vector representation of the input image after it has been extracted by the image encoder; The user instructions section explicitly specifies the requirements for model detection; The model response part (ASSISTANT) contains <think> 、 <conclusion> 、 <answer>Tags, <think>Labels are used to guide the model's reasoning for specific regions. <conclusion>Labels are used to guide the model's output of detection conclusions, describing the behavioral state of people in that area. <answer>The label is used to output the bounding box coordinates, followed by one or more bounding box coordinates in json format.
[0057] The design of this text format enables the visual language large model to more accurately understand the task intent and generate structured output through a unified question and answer structure and visual language; the identification of multiple targets in the image and the context explanation are realized by combining the bounding box coordinates and the natural language description; and the alignment of visual information and language description helps the visual language large model to learn a stronger visual-language joint representation in end-to-end training.
[0058] 2. Training scheme.
[0059] In the training, the structured text instruction and the reward mechanism containing positioning accuracy are used to enhance the inference and analysis capability of the model on the visual input, thereby effectively guiding the optimization process.
[0060] For example, the visual language large model can select Qwen2.5-VL-7B, Qwen is the model name (Qianwen), 2.5 is the version number, VL represents that the model has visual language (Vision-Language) capability, and 7B (7 billion) is the parameter quantity. Although Qwen2.5-VL-7B has certain ability in the target detection task by predicting the bounding box, the original output has problems such as repeated answer format and insufficient positioning accuracy. Therefore, the inference capability of the model is activated and strengthened from scratch, so that the model can autonomously construct a logical and clear thinking chain before generating the final answer. In order to guide the model to iterate in a more optimal direction, a set of refined reward mechanisms are designed. The input text instruction requires the model to analyze the image and generate a structured inference process, and then output the final conclusion and answer according to the predefined format. The reward mechanism evaluates the format and accuracy of the model output, and guides the subsequent optimization process accordingly.
[0061] (1) Reward mechanism.
[0062] The reward mechanism aims to enhance the inference process of the visual language large model and standardize its output. The present application uses two types of reward functions: format reward and accuracy reward. The format reward includes label format reward, bounding box format reward and non-repetition format reward, which are used to impose constraints on the inference process and output format, guide the visual language large model to generate a structured inference process, and suppress redundant or repetitive answers to ensure that the generated content meets the pre-set logic and expression specifications; the accuracy reward includes multi-target IoU reward and L1 reward, which quantitatively evaluate the consistency between the prediction results and the true labels based on IoU and L1 distance, etc. to improve the accuracy of positioning and identification, and enhance the overall perception ability of the model in multi-target scenarios. The above rewards are used to drive reinforcement learning.
[0063] Tag format rewards. If the visual language large model is... <think> and< / think> The output reasoning process between tags, in <conclusion> and< / conclusion> Output reasoning conclusions between tags, and in <answer> and< / answer> If the bounding box answer is output between labels, the sample is given a format reward of 1; otherwise, it is 0. The label format reward ensures that the output strictly follows the predefined structured format.
[0064] Bounding box format reward. The predicted bounding box answers of the visual language large model must be output in the form of a list in JSON format. That is, the format of the model output answer is restricted to [{'bbox': [ x_min , y_min , x_max , y_max]},…]. If the output meets the format requirements, the reward is 1; otherwise, it is 0.
[0065] Non-repeating format reward. The reasoning process is segmented into sentences, and repetition is checked. If there are no repeated sentences, the reward is 1.
[0066] To evaluate localization accuracy, we used an accuracy reward consisting of an IoU reward and an L1 distance reward for each bounding box containing the predicted bounding box. samples Calculate its bounding box with the true labeled bounding box. The reward score is calculated based on the number of predicted bounding boxes M and the number of true labeled bounding boxes N. Since the number of predicted bounding boxes M may differ from the number of true labeled bounding boxes N, a many-to-many matching problem needs to be addressed. Therefore, the Hungarian Algorithm is used to find the optimal one-to-one assignment, thus assigning a unique match to each prediction-label pair.
[0067] Bounding box IoU reward. After obtaining the best match, calculate the IoU of each pair of matching boxes. For any pair of matching boxes, if their IoU is greater than or equal to 0.5, increase the reward. This ensures that the total reward is within a reasonable range and is robust to different target numbers.
[0068] Bounding box L1 reward. Also based on the one-to-one correspondence obtained from the Hungarian algorithm, the L1 distance (in pixels) between the center points of each pair of matching boxes is calculated. If the L1 distance is less than or equal to a threshold (10 pixels), the reward is increased. .
[0069] 2. Optimization method.
[0070] GRPO is an on-policy reinforcement algorithm. For each input x, the policy of the large visual-language model before fine-tuning in the previous training round is applied. Generate a set of rollouts. Calculate each using the reward function The reward, to receive the reward And the relative advantage was calculated: ; GRPO optimizes the policy G of the large visual-language model after fine-tuning in the previous training rounds based on the following formula: .
[0071] The meanings of the parameters involved in the formula can be found in the previous introduction, and will not be repeated here.
[0072] Given an input image Text instructions The visual language big model generates a reasoning process, and then produces... The bounding box of the corresponding target object Rewards are calculated based on the reward mechanism described above, and these rewards jointly guide the optimization process by enhancing structural correctness and multi-object recognition performance. Under this reward mechanism, visual perception tasks can be handled better after training with GRPO reinforcement learning.
[0073] 3. Reasoning process.
[0074] After fine-tuning the large visual language model based on the above training scheme, the input image can be... Text instructions Input the finely tuned visual language model and guide it through the reasoning process.
[0075] The fine-tuned visual language model receives the input image. Text instructions Then, an efficient visual encoder transforms the image into a set of feature vectors, capturing not only the semantic information of the image but also preserving key details to ensure the accuracy of subsequent processing. These feature vectors are projected into a language space to facilitate their integration with text instructions. Interaction is then conducted. The projected image features are then integrated with the text instructions. This process involves complex interaction patterns, deeply combining image information with language understanding. The model can reason based on safety production standards, thereby generating more accurate and context-sensitive output. Mapping image features to the language space significantly enhances the model's understanding of the overall scene, enabling it to go beyond object recognition and provide in-depth scene analysis and explanation, such as... Figure 3 The image shows an example of the reasoning process of the fine-tuned visual language big model.
[0076] III. Effect Verification.
[0077] To demonstrate the performance of the above-described solutions in the embodiments of the present invention, experiments were conducted on a batch of industrial production safety data that had been collected and labeled.
[0078] 1. Experimental details.
[0079] In the experiment, 1585 images were used. For LoRA (Lower-Rank Adaptive) and full-parameter fine-tuning, text descriptions were used for supervised training of the model on text output. For reinforcement learning fine-tuning, bounding box coordinates were used to calculate the reward function and optimize the model, without text descriptions. The training and test sets were split in an 8:2 ratio. Qwen2.5-VL-7B was used as the large visual language model, trained using the DeepSpeed library. During training, a total batch size of 8 was used, with 8 samples per training step. The initial learning rate was set to 1e - 6, and the weight decay was 0.01.
[0080] 2. Evaluation indicators.
[0081] In terms of evaluation metrics, AP50 and AR10 were selected to evaluate the model's performance on the object detection task.
[0082] AP50 represents the average precision at an IoU threshold of 0.5. Specifically, it measures the model's overall ability to correctly locate targets by calculating the area under the curve of precision versus recall at different confidence thresholds.
[0083] AR10 (Average Recall@10) is defined as the average proportion of all real targets successfully recalled under the condition that a maximum of 10 prediction results are allowed to be output per image. It is applicable to multi-target scenarios.
[0084] Bleu1, Bleu2, Bleu3, Bleu4, METEOR, ROUGE-L, and CIDEr were selected to evaluate performance on natural language generation tasks. These evaluations compare the similarity between the generated text and the labeled text. Bleu (Bilingual Evaluation Understudy) evaluates the degree of n-gram matching between the generated and reference texts. METEOR considers synonyms, lexical variations, and word order. ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation) measures the longest common subsequence length (LCS) between the generated and reference summaries. Recall and precision are obtained based on the LCS, and then the F1 score is calculated. CIDEr (Consensus-based Image Description Evaluation) calculates the average similarity between the candidate sentence and all reference sentences.
[0085] The calculation of the above indicators all follow the official COCO evaluation standards.
[0086] 3. Experimental data and visualization results.
[0087] To investigate the impact of model training strategies on industrial unsafe behavior detection tasks, four different fine-tuning methods were compared: No fine-tuning: directly using a pre-trained large visual language model for inference without any parameter updates; LoRA fine-tuning: using the LoRA method to update only a small number of trainable parameters; Full parameter fine-tuning: performing end-to-end optimization of all model parameters; Reinforcement learning fine-tuning: iteratively optimizing the model based on rewards within the above reinforcement learning framework.
[0088] (3.1) Target detection performance.
[0089] The target detection performance is shown in Table 1. With the gradual enhancement of the fine-tuning strategy, the model performance is significantly improved.
[0090] Table 1: Target detection performance of Qwen2.5-VL-7B under different fine-tuning methods Fine-tuning method AP50 AR10 Qwen2.5-VL-7B 14.3 24.1 Lora fine-tuning 54.9 43.3 Full parameter fine-tuning 59.6 50.4 Reinforcement Learning Fine-tuning 70.8 73.7
[0091] On the AP50, the score without fine-tuning is only 14.3, indicating that the original model has limited localization capabilities in complex scenes; while LoRA fine-tuning improves it to 54.9, indicating that LoRA can effectively capture key feature changes; full parameter fine-tuning reaches 59.6, further illustrating the importance of fully optimizing model parameters; the best performance comes from reinforcement learning fine-tuning, with an AP50 as high as 70.8, which is significantly better than other methods, reflecting that the optimization path guided by the reward mechanism is more in line with the task objectives (such as structured reasoning, format standardization, etc.).
[0092] On the AR10 metric, the trend is consistent: without fine-tuning, the score is only 24.1, which is insufficient to effectively recall true anomalies; LoRA fine-tuning improves to 43.3, showing that it has strong generalization ability while maintaining lightweight design; full parameter fine-tuning reaches 50.4, demonstrating the advantage of full parameter space optimization; reinforcement learning fine-tuning achieves the best score of 73.7, indicating that it not only improves localization accuracy but also enhances the model's recall ability in multi-object scenarios.
[0093] The fine-tuning of reinforcement learning achieved optimal performance across all metrics, demonstrating its effectiveness in guiding models to generate high-quality inference and accurate detection. By introducing a task-oriented reward mechanism, reinforcement learning achieves deeper optimization, making it particularly suitable for tasks requiring structured output and high-precision localization.
[0094] (3.2) Natural language generation performance.
[0095] Regarding text output, Table 2 presents the evaluation results of the reinforcement learning fine-tuning strategy for Qwen2.5-VL-7B, which was not fine-tuned.
[0096] Table 2: Natural Language Generation Metrics of Qwen2.5-VL-7B under Reinforcement Learning index Qwen2.5-VL-7B Reinforcement Learning Fine-tuning Bleu1 31.8 37.1 Bleu2 9.3 28.7 Bleu3 3.8 22.8 Bleu4 0 18.1 METEOR 26.3 36 ROUGE L 29.2 48.2 CIDEr 0.4 18.8
[0097] The reinforcement learning strategy outperforms the original Qwen2.5-VL-7B model across the board, especially showing significant improvements in metrics such as ROUGE-L and METEOR, which measure semantic overlap. This indicates that by introducing a reward mechanism oriented towards bounding box accuracy, the model can indirectly promote consistency between generated descriptions and image content without interfering with its original language capabilities. In particular, when generating structured reasoning processes, reinforcement learning fine-tuning enables the model to better follow predefined formats, avoid repetition, and produce higher-quality text.
[0098] (3.3) Visualization results.
[0099] The inputs and outputs of Qwen2.5-VL-7B are as follows: Figure 4 The input and output of the model trained by reinforcement learning are as follows: Figure 5 The model first performs semantic parsing of the scene through multimodal understanding in structured thinking, identifies key objects and their spatial relationships, summarizes unsafe behaviors in the conclusion section, and outputs accurate coordinates in the answer section.
[0100] Furthermore, during testing of Qwen2.5-VL-7B, repetitive looping text was discovered, such as: **Not wearing a safety rope:** The worker in the picture is not wearing a safety rope, which is extremely dangerous when working at heights. A safety rope can prevent falls.
[0101] **Not wearing a safety rope:** The worker in the picture is not wearing a safety rope, which is extremely dangerous when working at heights. A safety rope can prevent falls.
[0102] **Not Wearing a Safety Rope:** The worker in the picture is not wearing a safety rope, which is extremely dangerous when working at heights. A safety rope can prevent falls. ...... The Qwen2.5-VL-7B model suffers from repetitive response formats. This issue is significantly improved in the model incorporating reinforcement learning, demonstrating that avoiding repetitive response formats through a reward mechanism—specifically, setting rewards for unique formats—plays a crucial role in enhancing the diversity and quality of the model's output.
[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0104] Example 2 This invention also provides an industrial unsafe behavior detection system, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 6 As shown, the system mainly includes: The model fine-tuning unit is used to collect labeled image data of industrial production safety scenarios and combine it with text instructions in a structured multimodal question-and-answer text format. It uses reinforcement learning algorithms to fine-tune the visual language model, including: inputting image data and corresponding text instructions into the visual language model, obtaining the response text of the visual language model, calculating the reward based on the format of the response text and the position of the predicted bounding box and the labeled true bounding box in the response text, and combining the reward calculation to optimize the objective function to fine-tune the visual language model. The industrial unsafe behavior detection unit, after fine-tuning, inputs text commands and the image to be detected into the fine-tuned visual language model, guiding the visual language model to output a structured multimodal question-and-answer text format response.
[0105] Since the main technical details of the above system have been described in detail in the previous embodiments, they will not be repeated here.
[0106] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.
[0107] Example 3 The present invention also provides a processing device, such as Figure 7 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.
[0108] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.
[0109] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example: Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc. The output device can be a display terminal; The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.
[0110] Example 4 The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.
[0111] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.
[0112] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.< / answer> < / conclusion> < / think> < / answer> < / conclusion> < / think>
Claims
1. A method for detecting unsafe industrial behaviors, characterized in that, include: Annotated image data of industrial production safety scenarios are collected, and text instructions in a structured multimodal question-and-answer text format are combined with reinforcement learning algorithms to fine-tune a large visual language model. The process includes: inputting image data and corresponding text instructions into the large visual language model to obtain the response text of the large visual language model; calculating the reward based on the format of the response text and the position of the predicted bounding box and the labeled true bounding box in the response text; and combining the reward calculation to optimize the objective function and fine-tune the large visual language model. After fine-tuning, the text instructions and the image to be detected are input into the fine-tuned visual language model, which then guides the visual language model to output a structured multimodal question-and-answer text format response.
2. The method for detecting unsafe industrial behavior according to claim 1, characterized in that, The structured multimodal question-and-answer text format includes the following text instructions: placeholders for image features, used to insert image features extracted by the visual language big model; security specification description text; and instruction text to guide the visual language big model in reasoning. Furthermore, the model response of the visual language big model is set to include three parts: reasoning process, reasoning conclusion, and predicted bounding box answer, and each part has corresponding labels at the beginning and end.
3. The method for detecting unsafe industrial behavior according to claim 2, characterized in that, The calculation of the reward based on the format of the response text and the positions of the predicted bounding box and the labeled true bounding box in the response text includes: The format reward is calculated, including label format reward, bounding box format reward and non-repeating format reward. The three rewards are combined as the format reward, which is used to constrain the format of the visual language model's response text and guide the visual language model to generate structured multimodal question-and-answer text format responses. Accuracy bonuses include: calculating various metrics based on the position of the bounding box and the ground truth bounding box, and determining the accuracy bonus based on the corresponding metric values.
4. The method for detecting unsafe industrial behavior according to claim 3, characterized in that, The label format reward includes: if the visual language model's response text outputs the reasoning process, reasoning conclusion, and predicted bounding box answer according to the label format set in the model response section, the reward value is 1; otherwise, it is 0. The bounding box format reward includes: if the format of the predicted bounding box answer in the response text of the visual language large model is the format set in the model response part, the reward value is 1, otherwise it is 0; The non-repetition format reward includes: segmenting the reasoning process in the response text of the visual language large model into sentences and detecting whether there are repetitions; if there are no repetitions, the reward value is 1, otherwise it is 0.
5. The method for detecting unsafe industrial behavior according to claim 3, characterized in that, The accuracy reward includes: multi-target IoU reward and L1 reward; Among them, the multi-objective IoU reward uses the IoU between the predicted bounding box and the ground truth bounding box as the metric, and the multi-objective IoU reward is determined based on the metric value; the L1 reward uses the L1 distance between the center point of the bounding box and the center point of the ground truth bounding box as the metric, and the L1 reward is determined based on the metric value.
6. The method for detecting unsafe industrial behavior according to claim 5, characterized in that, Let the predicted bounding box answer be denoted as ;in, Let M be the i-th predicted bounding box, and M be the total number of predicted bounding boxes; let the ground truth bounding box be denoted as . ,in, For the first There are N true bounding boxes; the Hungarian algorithm is used to find the optimal one-to-one assignment, assigning a unique matching relationship between each predicted bounding box and a true bounding box. When calculating the multi-objective IoU reward, for any matching predicted bounding box and ground truth bounding box, if the IoU exceeds a set first threshold, the reward is increased. The max function outputs the maximum value within the parentheses. When calculating the L1 reward, if the L1 distance between any matched predicted bounding box and the ground truth bounding box is less than a set second threshold, then the reward is increased. .
7. A method for detecting unsafe industrial behavior according to any one of claims 1 to 6, characterized in that, The objective function optimized by combining reward calculation includes: Let the strategy of the large visual language model before the fine-tuning in the last training round be denoted as... The output answer text is denoted as The corresponding reward is recorded as ,in, For the k-th answer text, The reward for the k-th response text, where K is the number of response texts, is calculated using the following formula: ； in, It represents the relative advantage of the k-th response text; mean(.) is the mean function, and std(.) is the standard deviation function; Let G be the policy of the large visual language model after the fine-tuning in the previous training round, and calculate the optimization objective function using the following formula: ； Where E represents the expectation, and This refers to strategy G and Use input x to perform reasoning and output the answer text. The conditional probability distribution; input x refers to text instructions and image data; clip is the truncation operation. The cutoff range parameter; The multiplication coefficients of the KL divergence; For reference strategy; It is policy G and reference policy The KL divergence between them.
8. An industrial unsafe behavior detection system, characterized in that, To implement the method according to any one of claims 1 to 7, comprising: The model fine-tuning unit is used to collect labeled image data of industrial production safety scenarios and combine it with text instructions in a structured multimodal question-and-answer text format. It uses reinforcement learning algorithms to fine-tune the visual language model, including: inputting image data and corresponding text instructions into the visual language model, obtaining the response text of the visual language model, calculating the reward based on the format of the response text and the position of the predicted bounding box and the labeled true bounding box in the response text, and combining the reward calculation to optimize the objective function to fine-tune the visual language model. The industrial unsafe behavior detection unit, after fine-tuning, inputs text commands and the image to be detected into the fine-tuned visual language model, guiding the visual language model to output a structured multimodal question-and-answer text format response.
9. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.
10. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.