Reinforcement learning method for visual language generative anomaly analysis
By training a visual language generative model using reinforcement learning algorithms, the problems of insufficient generalization and specialization in anomaly detection in existing technologies are solved. This enables accurate interpretation of surveillance footage and annotation of anomaly locations, thereby improving the anomaly analysis capabilities of the surveillance system.
Patent Information
- Application Number
- CN202510905869.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2025-06-29
- Filing Date
- 2025-07-02
- Publication Date
- 2025-11-21
AI Technical Summary
Existing visual language generative large models suffer from insufficient generalization and specialization when facing anomaly detection in complex monitoring scenarios. They are unable to accurately identify and locate abnormal events and cannot meet the needs of locating and tracing the source of abnormal events in real-world scenarios.
A visual language generative model is trained using reinforcement learning algorithms. By designing a reward function and iteratively optimizing it, and combining anomaly detection and bounding box generation for visual tasks, the model achieves accurate interpretation of surveillance footage and annotation of anomaly locations.
It enables efficient identification and location of abnormal events in monitoring footage, provides detailed explanations, enhances the monitoring system's ability to analyze anomalies in complex scenarios, and supports tasks such as security management and traffic dispatching.
Smart Images
Figure 4UFIMMT9LHBFTXISVOD9ONZGOBMJH2LKNHVLYPKK 
Figure 5LWXAQWQMT9RZD0U7T3684RT9LKGHR1HXWRL9L7L 
Figure 8525HVTOKKGFENO9DBR8JCK0ZKM6SQB33JFDB3N6
Abstract
Description
Technical Field
[0001] This invention relates to a visual language generation method, and more specifically to a reinforcement learning method for visual language generative anomaly analysis. Background Technology
[0002] With the increasing demand for intelligent security checks, visual monitoring systems based on surveillance cameras are widely used in construction site safety supervision and urban traffic management. However, abnormal events in these scenarios are diverse and complex, making it difficult for traditional anomaly detection methods based on fixed rules or single models to effectively address them. Although the emergence of large-scale visual language generative models has provided a new approach to anomaly analysis, their lack of adaptive training for complex scene rules results in insufficient generalization and specialization when facing ambiguous, dynamic, or complex anomalies in real-world monitoring scenarios. Furthermore, existing technologies often only determine whether an anomaly has occurred, but cannot accurately label the specific location of the anomaly in the image, failing to meet the needs of locating and tracing the source of anomalies in real-world scenarios. Therefore, a more efficient training method is urgently needed to enhance the ability of visual language generative models to understand and analyze anomalies under complex rules, enabling accurate interpretation of monitoring images and annotation of anomaly locations. Summary of the Invention
[0003] To address the shortcomings of existing technologies, this invention aims to provide a reinforcement learning algorithm for visual language generative anomaly analysis. Through innovative reinforcement learning training methods, it fully leverages the potential of large-scale visual language generative models, enabling them to adapt to diverse anomaly rules in complex monitoring scenarios such as construction sites and urban traffic. This algorithm not only accurately identifies and deeply analyzes abnormal events in monitoring footage but also provides detailed explanations of anomalies, helping managers quickly understand their nature. Furthermore, through optimized algorithm design, it achieves precise labeling of anomaly locations in monitoring footage, providing a reliable basis for subsequent anomaly handling, source tracing, and responsibility determination, effectively enhancing the anomaly analysis capabilities and application value of monitoring systems in complex scenarios.
[0004] To achieve the above objectives, the present invention provides the following technical solution: a reinforcement learning method for generative anomaly analysis of visual language, characterized by comprising the following steps: Step 1: Use other open-source visual language generative models to construct training data and tasks. The training data includes image-based question and rule generation, as well as task interpretation generation and rejection sampling processes. Step 2: Introduce reinforcement learning algorithms into the training process of the visual language generative model to train the model's ability to reason, interpret, and generate bounding boxes in this task. Step 3: Combine the anomaly detection of the visual task with the explanation content and the target box to design the reward function; Step four: Iteratively train the model based on the constructed data and the designed reward function, and then deploy it to the actual monitoring system after training is complete.
[0005] As a further improvement to the present invention, the specific steps for constructing training data and tasks using other open-source visual language generative models in step one are as follows: Step by step, using images collected from the monitoring scene or other open-source images as input data, VLM will perform deep visual analysis on each image and set corresponding anomaly detection rules based on elements such as objects, behaviors, and environment in the image content. In steps one and two, based on the set rules, VLM generates "yes" or "no" oriented questions. These questions precisely focus on possible anomalies, guiding subsequent answers to provide clear judgments. In these subsequent answers, VLM not only makes a "yes" or "no" judgment on the question but also provides a detailed explanation of the judgment result, explaining the basis for whether the anomaly has occurred or not. At the same time, VLM uses image coordinates to locate and mark the specific location of the anomaly in the image, with the top left corner of the image as the origin and described in pixels. Step 13: Integrate and encapsulate the generated questions and answers in JSON format to form standardized training data units.
[0006] As a further improvement of the present invention, the reinforcement learning algorithm in step two is a generalized relative policy optimization algorithm, and the optimization objective is expressed as follows: θ represents the policy parameters to be optimized, and θold represents the old policy parameters. Let represent the probability that the policy takes action a in state s. It is the relative advantage function. For the state-action value function, For the state value function, β represents the Kullback-Leibler divergence between the old and new policies, and β is a hyperparameter controlling the size of the trust domain.
[0007] As a further improvement of the present invention, the reward function in step three is as follows: ; Here, \(R_{detect}(s,a)\) represents the accurate reward for anomaly detection or rule violation, which can be understood as a classification task. If the classification is correct, the reward \(R_{detect}(s,a)\)=1 is obtained; otherwise, no reward is obtained \(R_{detect}(s,a)\)=0. The localization reward item \(R_{localize}(s,a)\) calculates the average intersection-union ratio between the predicted bounding box and the ground truth bounding box to evaluate the localization accuracy.
[0008] As a further improvement of the present invention, the reward function design steps in step three are as follows: Step 31: Calculate the IoU matrix between all predicted bounding boxes and ground truth bounding boxes; Step 32: Use a greedy algorithm to find the best matching pair in the IoU matrix, that is, the best match between each predicted box and the ground truth box; Step 33: Calculate the positional accuracy score based on the IoU value of the matching pair. If the IoU is greater than the set threshold, such as 0.5, the interpretation box is considered correct; otherwise, it is considered incorrect. Steps three and four: Finally, calculate the average IOU of the correctly interpreted boxes, where w1 and w2 are the weights of the two rewards.
[0009] As a further improvement to the present invention, the specific steps for model training and deployment in step four are as follows: Step 41: Input the training data into the training framework built on the GRPO algorithm according to the predetermined format. During the training process, the reward value calculated by the reward function is used to iteratively update the data in combination with the GRPO algorithm. Step 42: Training iteration. During the training iteration, cross-validation and other methods are used to evaluate the model performance in stages. The changing trends of key indicators such as anomaly detection accuracy, location annotation accuracy, and interpretation rationality are monitored to ensure that the model maintains good generalization ability on both the training and validation sets. When the model's performance indicators on the validation set reach the preset threshold and the training process tends to stabilize, the model training is completed. Step 43 involves deploying the trained visual language generative anomaly analysis model to servers or edge computing devices in real-world monitoring scenarios such as construction sites and urban traffic. This ensures that the model can respond quickly to real-time monitoring data streams, accurately identify abnormal events, provide immediate interpretation of monitoring footage, and automatically mark abnormal locations on the screen, thus providing efficient and reliable decision support for safety management and traffic scheduling.
[0010] The beneficial effects of this invention lie in its construction of a complete anomaly analysis catastrophe system based on reinforcement learning algorithms and visual language generative models, enabling efficient processing of anomalies in monitoring footage. Its core advantage lies in overcoming the limitations of traditional anomaly detection methods that merely determine the presence or absence of anomalies. It can accurately identify various open anomalies in scenarios such as construction sites and urban traffic, and clearly pinpoint anomaly areas in images through coordinate annotation, providing intuitive evidence for rapid location of anomalies. Simultaneously, the model can automatically generate detailed anomaly explanation text, combining scene rules to explain the cause and nature of the anomaly. This integrated solution of "identification + location + explanation" enhances the interpretability of monitoring systems, greatly improving the efficiency of managers' understanding and handling of anomalies, and providing strong technical support for safety management, traffic control, and other tasks in real-world scenarios. Detailed Implementation
[0011] The present invention will be further described in detail below with reference to the given embodiments.
[0012] This embodiment of a reinforcement learning method for visual language generative anomaly analysis includes the following steps: 1) Training Data and Task Construction: Training data and tasks are constructed using other open-source visual language generative models. The data includes image-based question and rule generation, as well as task interpretation generation and rejection sampling processes. 2) Application of reinforcement learning algorithms: Introducing reinforcement learning algorithms into the training process of visual language generative models to train the model's ability to reason, interpret, and generate bounding boxes in this task. 3) Reward Function Design: Based on anomaly detection in the visual task, and considering the explained content and the target bounding box, design the reward function. 4) Model training and deployment: The model is iteratively trained based on the constructed data and the designed reward function. After training is completed, it is deployed to the actual monitoring system for operation.
[0013] As a further explanation of the above steps, the details are as follows: S1 Training Data and Task Construction In the training data and task construction steps, the core is to use other Visual Language Models (VLMs) such as Qwen2-VL-72B and GPT-4O to transform images into training data in a question-and-answer format. The specific operation process is as follows: First, using images collected from monitoring scenes or other open-source images as input data, VLM performs deep visual analysis on each image. Based on elements such as objects, behaviors, and the environment in the image content, it sets corresponding anomaly detection rules. These rules can be arbitrary and do not need to conform to reality. For example, it can determine whether there is a violation based on people's clothing, behavior, or area markings, or whether there is a traffic anomaly based on traffic signs or vehicle driving status. For example, in a construction site scenario, workers are prohibited from wearing short sleeves, so the rule is: workers cannot wear short sleeves. Based on the set rules, VLM generates "yes" or "no" oriented questions. These questions must precisely focus on possible anomalies, guiding subsequent answers to provide clear judgments. In the answer generation stage, VLM is required not only to make a "yes" or "no" judgment on the question but also to provide a detailed explanation of the judgment result, explaining the basis for whether the anomaly has occurred or not. At the same time, VLM needs to use image coordinate positioning technology to accurately mark the specific location of the anomaly in the image. The coordinates are described in pixels with the upper left corner of the image as the origin. Finally, the generated questions and answers are integrated and packaged in JSON format to form standardized training data units. Each JSON data contains two parts: "question" and "answer," with the "answer" field further subdivided into... The answer field: This field is used to directly determine "yes" or "no" to the question. It is the core response to the question and provides a concise and clear conclusion as to whether an anomaly exists, making it easier for the subsequent model to quickly extract key judgment information. The `explanations` array stores detailed explanations of the judgment results. Multiple objects can be added to the array when there are multiple explanations or descriptions related to multiple anomalies. Each object contains the following subfields: The `bbox_2d` array records the location of anomalies within an image using two-dimensional bounding box coordinates. The array contains four values, representing the x and y coordinates of the top-left and bottom-right corners of the bounding box, respectively. Using the top-left corner of the image as the origin, precise coordinate positioning accurately identifies the specific area of the anomaly within the image, providing accurate data support for subsequent anomaly annotation and location. The label field provides a detailed description of the abnormal target within the bounding box, explaining the specific characteristics, nature, or manifestation of the abnormality. It supplements the abnormality information from a semantic level, helps the model understand the nature of the abnormality, and also facilitates manual verification and analysis.
[0014] For example, here is an example of a training set: { Question: "According to regulations, smoking is prohibited in no-smoking areas. Does the image in this picture violate this rule?" "answer": "Yes", "explanations": [ { "bbox_2d": [321, 183, 342, 203], Label: "There is a person in blue smoking on the left side of the corridor in the picture." } ] } S2 reinforcement learning algorithm application In the method for generative anomaly analysis of visual language proposed in this patent, the Generalized Relative Policy Optimization (GRPO) algorithm is used for model training. Based on the policy gradient framework, the GRPO algorithm effectively alleviates the problems of reward sparsity, high cost, and unstable training in traditional reinforcement learning by constructing a relative advantage function. The optimization objective of the GRPO algorithm can be expressed as: θ represents the policy parameters to be optimized, and θold represents the old policy parameters. This represents the probability that the policy will take action a in state s. Relative advantage function State-Action Value Function State value function Kullback-Leibler divergence between the old and new strategies β is a hyperparameter that controls the size of the trust region. S3 Reward Function Design In the application of the GRPO algorithm to visual language generative anomaly analysis in this patent, the reward mechanism is crucial. As the core feedback signal for policy optimization, it directly guides the model to learn anomaly analysis strategies that meet expectations. The reward is the numerical feedback obtained by the agent during its interaction with the environment, based on the anomaly analysis actions taken and the resulting outcomes. In the scenario of anomaly analysis in surveillance footage, if the model accurately identifies anomalies or reasonably interprets the anomalies and precisely labels the relevant locations, it will receive a positive reward; conversely, if there are misjudgments, ambiguous interpretations, or labeling deviations, a negative reward will be given. These reward values are incorporated into the relative advantage function. In the calculation, the quality of the action is evaluated by measuring the difference between the actual reward and the expected reward, which in turn affects the update direction of the policy network parameter θ. In this patent, to optimize the anomaly analysis capability of the visual language generative model in complex monitoring scenarios, a unique multi-dimensional reward function \(R(s,a)\) is designed, whose expression is: Here, \(R_{detect}(s,a)\) represents the accurate reward for anomaly detection or rule violation, which can be understood as a classification task. If the classification is correct, the reward \(R_{detect}(s,a)\) = 1; otherwise, it is not obtained and \(R_{detect}(s,a)\) = 0. The localization reward \(R_{localize}(s,a)\) calculates the average intersection-over-union (IoU) between the predicted bounding box and the ground truth bounding box to evaluate the localization accuracy. The specific steps are as follows: 1) Calculate the IoU matrix: First, calculate the IoU matrix between all predicted boxes and ground truth boxes.
[0015] 2) Greedy matching: The greedy algorithm is used to find the best matching pair in the IoU matrix, that is, the best match between each predicted box and the ground truth box.
[0016] 3) Calculate the location reward score: Calculate the location accuracy score based on the IoU value of the matching pairs. If the IoU is greater than a set threshold, such as 0.5, the interpreted bounding box is considered correct; otherwise, it is considered incorrect. Finally, calculate the average IoU of correctly interpreted bounding boxes. w1 and w2 are the weights of the two types of rewards.
[0017] S4 Model Training and Deployment After completing the construction of training data and tasks, the application of reinforcement learning algorithms, and the design of reward functions, the model training and deployment phase begins. First, the training data is input into the training framework built based on the GRPO algorithm according to a predetermined format. During training, the reward value calculated by the reward function is used for iterative updates in conjunction with the GRPO algorithm.
[0018] During the training iterations, cross-validation and other methods are used to periodically evaluate model performance, monitoring the changing trends of key indicators such as anomaly detection accuracy, location annotation precision, and interpretation rationality to ensure that the model maintains good generalization ability on both the training and validation sets. Model training is complete when the model's performance indicators on the validation set reach a preset threshold and the training process stabilizes. During the model deployment phase, the trained visual language generative anomaly analysis model is deployed to servers or edge computing devices in actual monitoring scenarios such as construction sites and urban traffic. This ensures that the model can respond quickly in real-time monitoring data streams, accurately identify abnormal events, provide immediate interpretation of monitoring images, and automatically mark abnormal locations on the images, providing efficient and reliable decision support for safety management, traffic scheduling, and other tasks.
[0019] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.
Claims
1. A reinforcement learning method for generative anomaly analysis of visual language, characterized in that: Includes the following steps: Step 1: Use other open-source visual language generative models to construct training data and tasks. The training data includes image-based question and rule generation, as well as task interpretation generation and rejection sampling processes. Step 2: Introduce reinforcement learning algorithms into the training process of the visual language generative model to train the model's ability to reason, interpret, and generate bounding boxes in this task. Step 3: Combine the anomaly detection of the visual task with the explanation content and the target box to design the reward function; Step four: Iteratively train the model based on the constructed data and the designed reward function, and then deploy it to the actual monitoring system after training is complete.
2. The reinforcement learning method for generative anomaly analysis of visual language according to claim 1, characterized in that: The specific steps for constructing training data and tasks using other open-source visual language generative models in step one are as follows: Step by step, using images collected from the monitoring scene or other open-source images as input data, VLM will perform deep visual analysis on each image and set corresponding anomaly detection rules based on elements such as objects, behaviors, and environment in the image content. In steps one and two, based on the set rules, VLM generates "yes" or "no" oriented questions. These questions precisely focus on possible anomalies, guiding subsequent answers to provide clear judgments. In these subsequent answers, VLM not only makes a "yes" or "no" judgment on the question but also provides a detailed explanation of the judgment result, explaining the basis for whether the anomaly has occurred or not. At the same time, VLM uses image coordinates to locate and mark the specific location of the anomaly in the image, with the top left corner of the image as the origin and described in pixels. Step 13: Integrate and encapsulate the generated questions and answers in JSON format to form standardized training data units.
3. The reinforcement learning method for generative anomaly analysis of visual language according to claim 1 or 2, characterized in that: The reinforcement learning algorithm in step two is the generalized relative policy optimization algorithm, and the optimization objective is expressed as follows: θ represents the policy parameters to be optimized, and θold represents the old policy parameters. Let represent the probability that the policy takes action a in state s. It is the relative advantage function. For the state-action value function, For the state value function, β represents the Kullback-Leibler divergence between the old and new policies, and β is a hyperparameter controlling the size of the trust domain.
4. The reinforcement learning method for generative anomaly analysis of visual language according to claim 1 or 2, characterized in that: The reward function in step three is as follows: ； Here, \(R_{detect}(s,a)\) represents the accurate reward for anomaly detection or rule violation, which can be understood as a classification task. If the classification is correct, the reward \(R_{detect}(s,a)\) = 1 is obtained; otherwise, no reward is obtained \(R_{detect}(s,a)\) = 0. The localization reward item \(R_{localize}(s,a)\) evaluates the location accuracy by calculating the average intersection-union ratio between the predicted bounding box and the ground truth bounding box.
5. The reinforcement learning method for generative anomaly analysis of visual language according to claim 4, characterized in that: The steps for designing the reward function in step three are as follows: Step 31: Calculate the IoU matrix between all predicted bounding boxes and ground truth bounding boxes; Step 32: Use a greedy algorithm to find the best matching pair in the IoU matrix, that is, the best match between each predicted box and the ground truth box; Step 33: Calculate the positional accuracy score based on the IoU value of the matching pair. If the IoU is greater than the set threshold, such as 0.5, the interpretation box is considered correct; otherwise, it is considered incorrect. Steps three and four: Finally, calculate the average IOU of the correctly interpreted boxes, where w1 and w2 are the weights of the two rewards.
6. The reinforcement learning method for generative anomaly analysis of visual language according to claim 1 or 2, characterized in that: The specific steps for model training and deployment in step four are as follows: Step 41: Input the training data into the training framework built on the GRPO algorithm according to the predetermined format. During the training process, the reward value calculated by the reward function is used to iteratively update the data in combination with the GRPO algorithm. Step 42: Training iteration. During the training iteration, cross-validation and other methods are used to evaluate the model performance in stages. The changing trends of key indicators such as anomaly detection accuracy, location annotation accuracy, and interpretation rationality are monitored to ensure that the model maintains good generalization ability on both the training and validation sets. When the model's performance indicators on the validation set reach the preset threshold and the training process tends to stabilize, the model training is completed. Step 43 involves deploying the trained visual language generative anomaly analysis model to servers or edge computing devices in real-world monitoring scenarios such as construction sites and urban traffic. This ensures that the model can respond quickly to real-time monitoring data streams, accurately identify abnormal events, provide immediate interpretation of monitoring footage, and automatically mark abnormal locations on the screen, thus providing efficient and reliable decision support for safety management and traffic scheduling.