A threat object detection method based on reinforcement learning

Abstract

The application discloses a threat object detection method based on reinforcement learning, and aims to improve the threat object detection accuracy in the field of automatic driving. The technical scheme is to construct a threat object reasoning and detection system based on reinforcement learning, which is composed of a strategy model, a reward value calculation module, a group advantage calculation module and a strategy model updating module. The strategy model in the system is trained to obtain a trained threat object reasoning and detection system with threat reasoning and threat object perception capabilities. Finally, the trained threat object detection system is used for threat object detection to obtain the boundary box and threat level of the threat object. The application enables the multi-modal large language model to realize fine-grained object-level threat reasoning based on the context information of the driving scene, significantly improves the understanding ability of the automatic driving system to threat semantics and the perception ability of the threat object, and improves the threat object detection accuracy in the field of automatic driving.

Description

Technical Field

[0001] This invention relates to the field of threat object detection in autonomous driving, specifically to a threat object detection method based on reinforcement learning, which aims to improve the accuracy of threat object detection in the field of autonomous driving. Background Technology

[0002] Object detection, a core task of autonomous driving perception systems, has evolved from basic category recognition to complex scene understanding. Traditional detection methods primarily focus on identifying finite sets of categories (such as vehicles and pedestrians), achieving significant results but still facing serious challenges in real-world driving scenarios. In recent years, with the emergence of new research directions such as out-of-distribution (OOD) and open-world object detection (OWOD), researchers have begun to explore the ability to identify rare objects and unknown obstacles. These technological breakthroughs have significantly improved the ability of autonomous driving perception systems to perceive unknown objects.

[0003] However, current mainstream detection methods have a fundamental limitation: they rely excessively on predefined category semantics for target recognition, neglecting key cognitive characteristics in the human driving decision-making process. In actual driving, drivers' attention allocation exhibits a clear threat-oriented characteristic—they focus more on dynamic factors that may genuinely affect driving safety (such as objects suddenly entering the lane, vehicles with abnormal driving states, etc.), rather than mechanically scanning all objects belonging to a specific category. This semantic category-based detection method differs fundamentally from the risk assessment mechanism of human drivers, making it difficult for autonomous driving perception systems to respond to potential hazards in complex traffic scenarios in a way that aligns with human expectations.

[0004] More specifically, existing detection models in autonomous driving perception systems lack the ability to understand the behavioral intentions of traffic participants and cannot assess the threat level of objects under different circumstances. For example, the impact of a pedestrian standing on the roadside versus crossing the road on driving decisions is drastically different; similarly, the threat level of a vehicle traveling in its lane versus merging with another vehicle is completely different. This cognitive gap prevents traditional detection systems in autonomous driving perception systems from providing perception results that truly meet driving safety requirements. Therefore, developing intelligent detection frameworks with threat perception capabilities and learning model reasoning abilities from visual perception to risk assessment is crucial for improving the safety of autonomous driving systems.

[0005] However, in driving scenarios, "threat" is a complex concept with a high degree of context dependence. Unlike traditional category recognition, threat assessment often requires multi-level cognitive reasoning. This necessitates that detection models can simulate the judgment process of human drivers, comprehensively analyzing various factors such as the spatial location, motion state, physical characteristics, and environmental occlusion relationships of target objects. Given this complexity, simple semantic learning methods are insufficient to achieve accurate understanding and interpretable analysis of threats.

[0006] The paper "Lisa: Reasoning segmentation via large language model" by Lai, Xin, et al., presented at the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2024, proposes a multimodal large language model-based collaborative perception model to understand the implicit semantics of human language and use reasoning techniques to perceive quasi-object positions. However, this method cannot generate content for reasoning analysis, thus reducing its reasoning ability. The paper "Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement" by Liu, Yuqi, et al., presents Seg-zero, a reasoning-chain guided segmentation method using cognitive reinforcement. This method is the first to apply the reinforcement learning algorithm GRPO (Group Relative Policy Optimization) to reasoning perception tasks. <think>The `<think>` tag allows multimodal large language models to output thought content within the tag, enhancing their inherent reasoning ability. However, this method can only perceive a single object and cannot perceive multiple objects. The paper "Liu, Yuqi, et al. VisionReasoner: Unified Visual Perception and Reasoning via Reinforcement Learning." arXiv preprint arXiv:2505.12081(2025). Liu, Yuqi, et al.'s paper, VisionReasoner: Unified Visual Perception and Reasoning via Reinforcement Learning, utilizes a binary matching algorithm based on the Seg-zero method to achieve multi-object reasoning perception. However, this method can only perform reasoning analysis at the image level and cannot perform reasoning analysis for each object individually.

[0007] The aforementioned reasoning and perception methods based on multimodal large language models have achieved significant progress in conventional human-computer interaction scenarios. However, reasoning analysis and collaborative perception technologies in the field of autonomous driving are still in their infancy. Compared to the varied questioning needs of humans in conventional scenarios, the goal of reasoning analysis in autonomous driving scenarios is consistent: to reason about and identify objects that pose a potential threat to driving. However, "threat" is a vague semantic term, requiring models to have stronger contextual understanding and reasoning capabilities, as this involves simulating the threat assessment process during human driving. Furthermore, in the field of autonomous driving, the key to threat detection lies in performing independent and differentiated semantic analysis on each target object. However, existing general-domain reasoning and perception methods mainly rely on image-level global text prompts for reasoning. Even if multiple detection targets can be output, these objects are still at the same semantic level (e.g., detecting the part of a plant that absorbs sunlight). This is significantly different from the needs of actual driving scenarios. The threat semantics of different traffic participants (such as pedestrians, vehicles, obstacles, etc.) are fundamentally different (e.g., a pedestrian crossing the road normally at a traffic light versus a pedestrian suddenly crossing the road), requiring separate threat reasoning analysis for each object in conjunction with the specific scenario.

[0008] Improving the threat analysis, reasoning, and perception capabilities of autonomous driving perception systems for traffic objects remains a technical issue of great concern to those skilled in the art. Summary of the Invention

[0009] The technical problem this invention addresses is a key issue in intelligent perception within the field of autonomous driving: how to improve the reasoning, analysis, and perception capabilities of autonomous driving systems for threatening objects, thereby enhancing the accuracy of threatening object detection in autonomous driving. To overcome the limitations of existing detection methods, which only perform semantic classification at the image level and lack contextual reasoning capabilities, this invention proposes a reinforcement learning-based threatening object detection method. This method utilizes a multimodal large language model and the GRPO reinforcement learning framework to endow the autonomous driving system with human-like threat assessment thinking capabilities. It also designs an unsupervised object-level thinking reward mechanism, enabling the multimodal large language model to achieve fine-grained object-level threat reasoning based on contextual information from specific driving scenarios. This significantly improves the autonomous driving system's understanding of threat semantics and its perception of threatening objects, thereby enhancing the accuracy of threatening object detection in the autonomous driving field.

[0010] To address the aforementioned technical problems, the technical solution of this invention is to construct a threat object reasoning and detection system based on reinforcement learning. This system comprises a policy model, a reward value calculation module, a group advantage calculation module, and a policy model update module. The dataset required for the reinforcement learning-based threat object reasoning and detection system is prepared and constructed, and then divided into a training set and a test set. The training set is used to train the multimodal large language model employed by the policy model in the reinforcement learning-based threat object reasoning and detection system, resulting in a trained threat object reasoning and detection system with threat reasoning and threat object perception capabilities. Finally, the trained threat object detection system is used to detect threat objects in user-input images, obtaining the bounding boxes and threat levels of the threat objects.

[0011] This invention includes the following steps:

[0012] The first step is to construct a threat object reasoning and detection system based on reinforcement learning. This system consists of a policy model, a reward value calculation module, a group dominance calculation module, and a policy model update module. The reward value calculation module, group dominance calculation module, and policy model update module are only used during policy model training.

[0013] The policy model is a multimodal large language model connected to the reward value calculation module and the policy model update module. The policy model receives user input images and prompt text, and performs multimodal information processing on the input images and prompt text. A multimodal large language model needs to have multimodal processing capabilities and support bounding box detection, such as Qwen2.5VL-7B (see the paper "Bai, Shuai, et al. Qwen2.5-vl technical report." arXiv preprint arXiv:2502.13923(2025).). The network weight parameters of the multimodal large language model adopt VisionReasoner (see the paper "Liu, Yuqi, et al. VisionReasoner: Unified Visual Perception and Reasoning via Reinforcement Learning.arXiv preprintarXiv:2505.12081(2025). Liu, Yuqi et al.'s paper: VisionReasoner: Unified Visual Perception and Reasoning based on Reinforcement Learning"). VisionReasoner is an inference model that is retrained based on Qwen2.5VL-7B. During the training of the policy model, the policy model generates a batch of candidate output texts for the prompt text and image input by the user, obtains the output text set, and sends the output text set to the reward value calculation module. The policy model receives the optimized loss value of the updated policy model from the policy model update module, updates the policy model weights according to the optimized loss value, and obtains the trained policy model. When the trained policy model performs threat object detection on the prompt text and image input by the user, the policy model performs multimodal processing on the prompt text and image to obtain the output text. The text parsing module in the reward value calculation module receives the output text and performs text parsing to obtain image-level threat inference content, object-level bounding box prediction, object-level threat level prediction, and object-level threat inference content. The object-level bounding box prediction and object-level threat level prediction (high threat, medium threat, low threat) are used as the final output.

[0014] The reward value calculation module is connected to the policy model and the group advantage calculation module. During policy model training, the reward value calculation module calculates the reward value for each text in the output text set received from the policy model, forming a final reward value list. The reward value calculation module consists of a text parsing module, an output format reward value calculation module, a non-repeating output reward value calculation module, a prediction quantity reward value calculation module, and an adder. The text parsing module parses the output text set, extracting the image-level threat inference content and object-level response content for each output text, resulting in a parsed output text set. This parsed output text set is then sent to the output format reward value calculation module, the non-repeating output reward value calculation module, and the prediction quantity reward value calculation module. The object-level response content consists of object-level bounding box prediction, object-level threat level prediction, and object-level threat inference content. The output format reward value calculation module calculates the output format reward value for each output text in the output text set; the non-repeating output reward value calculation module calculates the non-repeating output reward value for each output text in the output text set; and the prediction quantity reward value calculation module calculates the prediction quantity reward value for each output text in the output text set. The adder sums the three reward values ​​for each output text in the above output text set to obtain the final reward value for each output text in the output text set. These final reward values ​​are placed in the final reward value list and sent to the group advantage calculation module.

[0015] The group advantage calculation module is connected to the reward value calculation module and the strategy model update module. The group advantage calculation module receives the final reward value list from the reward value calculation module, calculates the mean and standard deviation of all final reward values ​​in the list, and standardizes the list to obtain the advantage performance of each final reward value relative to the group. This advantage performance is then placed into a relative group advantage value list, which is sent to the strategy model update module.

[0016] The policy model update module is connected to the group dominance calculation module and the policy model. The policy model update module receives a reference model input by the user and calculates an optimization loss value based on the list of relative group dominance values ​​received from the group dominance calculation module. This optimization loss value is used for gradient backpropagation to update the weight parameters of the policy model. The optimization loss value calculation includes the Kullback-Leibler (KL) divergence loss between the policy model and the reference model, and the group dominance loss. The reference model and the policy model use the same multimodal large language model, which is an untrained multimodal large language model with frozen weight parameters (i.e., the weights of the reference model will not change). The KL divergence loss is used to penalize the deviation between the updated policy model and the reference model, thereby maintaining training stability; the group dominance loss is used to encourage the generation of text output with high reward values, which better aligns with human preferences.

[0017] The second step is to construct the training and test sets, using the following method:

[0018] 2.1 We collect rare class images from autonomous driving scenarios as a threat object inference and detection dataset. The method is as follows:

[0019] 2.1.1 A subset of the validation set (1000 images) of the autonomous driving scene dataset CODA (see Li, Kaican, et al. "Coda: A real-world road corner case dataset for object detection in autonomous driving. European conference on computer vision. Cham: Springer Nature Switzerland, 2022.") is used as the training dataset D for the threat object inference detection system. train Training dataset D train The number of images N T =1000; 2.1.2 A subset of the test set of the CODA autonomous driving scenario dataset (500 images in total), a subset of the Waymo autonomous driving dataset (see the paper "Sun, Pei, et al. Scalability inperception for autonomous driving: Waymo open dataset." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020." Sun, Pei et al.'s paper: Scalability inperception for autonomous driving: Waymo open dataset) (122 images in total), and the Argoverse 2 autonomous driving dataset (see the paper "Wilson, Benjamin, et al. Argoverse 2: Next generation datasets for self-driving perception and forecasting. arXiv preprint arXiv:2301.00493(2023." Wilson, Benjamin et al.'s paper: Argoverse 2) 2: A subset of the next-generation dataset for autonomous driving perception and prediction (201 images in total), a subset of the autonomous driving dataset KITTI (see "Geiger, Andreas, et al. Vision meets robotics: The kitti dataset. The international journal of robotics research 32.11(2013):1231-1237." Geiger, Andreas et al. paper: Vision meets robotics: kitti dataset.) (69 images in total), and a subset of the autonomous driving dataset Nuscenes (see "Caesar, Holger, et al. nuscenes: A multimodal dataset for autonomous driving. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020." Caesar, Holger et al. paper nuscenes: A multimodal dataset for autonomous driving.) (44 images in total) as a test dataset for threat object inference detection D. test D test It contains a total of M = 936 images.

[0020] 2.1.3 Due to inherent cognitive biases among individuals, it is difficult to arrive at a universally accepted label. For example, regarding traffic cones on the road, an aggressive driver might perceive them as a moderate threat, while a cautious driver might label them as a high threat; both judgments are reasonable. (Find the driver in position G regarding position D) test The M images in the dataset are manually annotated to obtain the manually annotated D. test Each manually labeled object includes the bounding box location of the threat object and its threat level (high threat, medium threat, low threat). G is a positive integer greater than 1.

[0021] 2.1.4 To further enhance the quality of annotation, the manually annotated D... test Samples with significant disagreements (labeled objects with threat level differences greater than one level) were reviewed, and a consensus was finally reached (labeled objects with threat level differences less than or equal to one level), resulting in the reviewed D. test , let it be DD test .

[0022] 2.2 For the training dataset D train Images and DD test The image is preprocessed using the following method:

[0023] 2.2.1 For the training dataset D train T images are scaled down to 840×840 pixels. The resulting preprocessed training dataset is denoted as D′. train .

[0024] 2.2.2 Regarding DD test M images were scaled down to 840×840 pixels. Meanwhile, DD... test The bounding box annotations in the dataset are scaled proportionally. This yields the preprocessed, reviewed test dataset, denoted as DD′. test .

[0025] The third step involves using the gradient backpropagation method to train the policy model on the reinforcement learning-based threat detection system constructed in the first step. This yields both the trained policy model and the trained reinforcement learning-based threat detection system. The method is as follows:

[0026] 3.1 Initialize the weight parameters of the policy model and reference model in the reinforcement learning-based threat object detection system. The weight parameters of the policy model and reference model are initialized using the weight parameters of the VisionReasoner model (an inference model retrained on the Qwen2.5VL-7B model).

[0027] 3.2 Setting Training Parameters for the Reinforcement Learning-Based Threat Detection System. AdamW (see "Loshchilov I, Hutter F. Decoupled Weight Decay Regularization [C] / / International Conference on Learning Representations." by Loshchilov I et al.) was selected as the model training optimizer. The learning rate was 0.000001, and the hyperparameter "weight decay" was 0.01. The batch size for network training was B = 16. The initial GRPO population sampling size was S = 8.

[0028] 3.3 The training of the threat object inference and detection system based on reinforcement learning is performed by calculating the final reward value list of the current output text set through the reward value calculation module, then calculating the relative group advantage value set through the group advantage calculation module, and finally calculating the optimized loss value and updating the policy model using the policy model update module until training is completed, resulting in the trained policy model. The specific method is as follows:

[0029] 3.3.1 Initialize batch sequence number N b =1. The strategy model reads the Nth bit of user input. b The prompt text in the batch is denoted as I. text I text The content includes the following prompts: Role prompt: "You are a professional driver with extensive driving experience."; Task prompt: "This is an image obtained from the vehicle's front camera. You need to detect the bounding boxes of threatening objects and estimate their threat level. Threat levels are categorized as high, medium, and low threat."; Output format prompt: "In..." <think>and< / think> The tag outputs image-level threat inference content, in <answer> and< / answer> The tag outputs object-level responses. The object-level responses are a JSON list. Each element contains a "think" key representing object-level threat inference, a "bbox_2d" key representing object-level bounding box prediction, and a "threat_level" key representing object-level threat level prediction. The number of elements in the list represents the number of predicted threat objects. The output format is: <think> Image-level threat reasoning content< / think> , <answer> [{"bbox_2d": object-level bounding box prediction, "think": object-level threat inference content, "threat_level": object-level threat level prediction},...]< / answer> ".

[0030] 3.3.2 The policy model is derived from the preprocessed training dataset D′ train Read the Nth b The batch contains B input images, which are represented as a matrix I of size B×H×W×3. train Where H represents the width of the input image, W represents the height of the input image, and "3" represents the three RGB channels of the image.

[0031] 3.3.3 Strategy Model for I train and I text Multimodal processing is performed to obtain the output text set I. output I output The output text has S group dominance sampling. Send to the reward value calculation module. The strategy model outputs according to the output format prompts required in 3.3.1, for example, outputting " <think> Image-level threat reasoning content< / think> , <answer> [{"bbox_2d": object-level bounding box prediction, "think": object-level threat inference content, "threat_level": object-level threat level prediction},...]< / answer> ".

[0032] 3.3.4 The reward value calculation module receives I output I is calculated using an unsupervised reward value calculation method. output The reward value corresponds to S output texts. The text parsing module parses the content of the output texts and outputs them as key-value pairs in JSON format, resulting in a parsed set of output texts. The output format reward value calculation module, the non-repeating output reward value calculation module, and the prediction quantity reward value calculation module simultaneously calculate the reward value for each of the S output texts. output The reward value is calculated from the S output texts, and then summed by an adder to obtain the final reward value, resulting in a list of S final reward values. This list is then sent to the group advantage calculation module. The specific method is as follows:

[0033] 3.3.4.1 The text parsing module receives I from the policy model output , to I output The text is parsed to obtain a set of parsed text. Will The method for sending the output format reward value calculation module is as follows:

[0034] 3.3.4.1.1 Initialize variable i = 1, and initialize the parsed text set. It is an empty set.

[0035] 3.3.4.1.2 Output the i-th text. Perform text parsing. Initialize. Parsed text output An empty JSON dictionary. Extract... middle <think> and< / think> The content between tags, as Image-level threat reasoning content. Extraction middle <answer> and< / answer> The content between tags, as The object-level response content is further parsed to obtain a JSON-formatted list, denoted as the object-level response list. Each list element consists of "bbox_2d" (object-level bounding box prediction), "think" (object-level threat inference content), and "threat_level" (object-level threat level prediction). The number of elements in the JSON-formatted list represents the number of predicted threat objects.

[0036] 3.3.4.1.3 If i ≤ S, let i = i + 1, then... deposit Go to 3.3.4.1.2; If i > S, then the parsed text set consists of S parsed texts. Go to 3.3.4.2.

[0037] 3.3.4.2 Output Format: The reward value calculation module receives data from the text parsing module. right The output format reward value is calculated for each parsed output text, using the following method:

[0038] 3.3.4.2.1 Initialize variable i = 1, initialize the output format reward value list It is an empty list.

[0039] 3.3.4.2.2 Output the i-th parsed text Computational thinking format reward value: If Correct match <think>and <answer>Tags, i.e. If neither the image-level threat reasoning nor the object-level response is empty, then let Mindset Reward Value Otherwise

[0040] 3.3.4.2.3 Output the i-th parsed text Calculate reward based on answer format. Extract the list of object-level answers. Calculate. Answer format reward

[0041]

[0042] in, This is the first indicator function. It returns 1.0 if "bbox_2d" can be effectively parsed from all elements in the object-level answer list, and 0.0 otherwise. This is the second indicator function. It returns 1.0 if "think" can be effectively parsed from all elements in the object-level answer list, and 0.0 otherwise. This is the third indicator function. It returns 1.0 if "think" can be effectively parsed from all elements in the object-level answer list, and 0.0 otherwise.

[0043] 3.3.4.2.4 Reward Values ​​for Thinking Format Rewards for answer format Add them together to get Output format reward value

[0044]

[0045] 3.3.4.2.5 If i ≤ S, let i = i + 1, then... deposit Go to 3.3.4.2.2; if i > S, then the result consists of S output format reward values. Will Send to the adder, then proceed to 3.3.4.3.

[0046] 3.3.4.3 The non-repeating output reward value calculation module receives from the text parsing module. right Each parsed output text in the model receives a non-repeating output reward value. Repetitive outputs often indicate a lack of thoughtful consideration in the model and may even produce illusory content. To address this issue, this invention introduces a non-repeating output reward, penalizing repeated inference text content and repeated object box outputs, thereby achieving more context-aware and differentiated analysis. The process of calculating the non-repeating output reward value is as follows:

[0047] 3.3.4.3.1 Initialize i = 1, and initialize a list of non-repeating reward values. It is an empty list.

[0048] 3.3.4.3.2 Calculate the non-repetitive reward value for the i-th image-level thought. from Extract image-level threat inference content from the text, and split the text into sentences to obtain the set T of clauses for the i-th image-level threat inference content. think , There are E clauses in total. Calculate the clause set T. think Non-repeating reward value R think-nr (T think The process is as follows:

[0049] Indicates traversal of T think All clause pairs in T (a clause pair refers to T) think Each element in the text is paired with another element in the text. Clauses and If the clauses are identical, increment the count by 1. If the final count is less than 2, then let... Image-level thinking non-repetitive reward value Otherwise

[0050] 3.3.4.3.3 Calculate the non-repeating reward value for the i-th bounding box. from Retrieve the object-level response list, then extract the content of the "bbox_2d" key from the object-level response list to obtain the bounding box set T of the bounding box prediction for the i-th object level. bbox , T bbox There are K object elements in total. Let T be the bounding box of the k-th predicted threat object. Calculate the bounding box set T. bbox Non-repeating reward value R bbox-nr (T bbox The process is as follows:

[0051]

[0052] |T bbox | represents T bbox The number of elements in the set, |unique(T) bbox )| represents T bbox The number of distinct elements in a set. If |T bbox |==|unique(T bbox )|, which means T bbox If all bounding boxes in the set are distinct, then let Non-repeating reward value of the object frame Otherwise

[0053] 3.3.4.3.4 Calculate the non-repeating reward value for the i-th object-level thinking. from Extract the object-level response list, then extract the "think" content from the object-level response list to obtain the threat reasoning content set T for the i-th object level. obj-think , There are K elements in total. Let T be the object-level inference content for the k-th predicted threat object. Calculate the threat inference content set T. obj-think Non-repeating reward value R objthink-nr (T obj-think The process is as follows:

[0054] Indicates traversal of T obj-think All object-level reasoning content text pairs (T) obj-think Each element in the text is paired with another element in the text. and If they are exactly the same, increment the count by 1. If the final count is less than 2, then let... Non-repetitive reward value for object-level thinking Otherwise

[0055] 3.3.4.3.5 Non-repetitive reward value for image-level thinking Non-repeating reward value of the object frame Object-level thinking non-repetitive reward value Add them together to get the i-th non-repeating output reward value.

[0056]

[0057] 3.3.4.3.6 If i ≤ S, let i = i + 1, then... deposit In section 3.3.4.3.2; if i > S, obtain S non-repeating output reward values. Will Send to the adder, then proceed to 3.3.4.4.

[0058] 3.3.4.4 The predicted quantity reward value calculation module receives data from the text parsing module. While rewarding the avoidance of duplicate outputs can improve the independence of object-level reasoning, it may also lead the model to output only one prediction box to avoid repetition, resulting in missed detections. Therefore, this invention introduces a prediction quantity reward to encourage the model to make multiple reasonable predictions. However, too many predictions may lead to excessively long outputs and affect the reasoning quality for each object. To balance this issue, this invention applies a flexible constraint on the number of predictions, incentivizing the policy model to improve prediction coverage within a reasonable range while maintaining the accuracy of reasoning for individual objects. The process of calculating the predicted quantity reward value is as follows:

[0059] 3.3.4.4.1 Initialize i = 1, initialize the prediction quantity reward value list An empty list.

[0060] 3.3.4.4.2 Calculate the reward value for the i-th predicted quantity. from Extract the object-level response list, then extract the "think" content from the object-level response list to obtain the threat reasoning content set T for the i-th object level. obj-think , There are K elements in total. This is the object-level inference content for the k-th predicted threat object. Since the prediction quantity reward only requires the number of predicted object boxes K to be calculated, the prediction quantity reward value R is calculated based on the number of predicted object boxes K. num The process for (K) is as follows:

[0061] N is the constraint inflection point parameter, set to 5. Then the reward value for the i-th predicted quantity...

[0062] 3.3.4.4.3 If i ≤ S, let i = i + 1, then... deposit In section 3.3.4.4.2; if i > S, obtain S non-repeating output reward values. Will Send to the adder, then proceed to 3.3.4.5.

[0063] 3.3.4.5 Adder Receiver Output Format: List of Reward Values Output a list of reward values ​​without duplicates. List of predicted quantity reward values Summing each of the three lists element by element yields a list of S total reward values, denoted as the final reward value list L. R , This is the i-th final reward value. Let L... R Send to the group advantage calculation module.

[0064] 3.3.5 The group advantage calculation module receives L from the reward value calculation module. R According to L R calculate The advantage value, obtained The corresponding advantage values ​​A1,…,A i ,…A S Send the advantage value list A to the policy model update module, where A = {A1, ..., A...} i ,…A S The method is:

[0065] 3.3.5.1 Initialize i = 1, and initialize the advantage value list A as an empty list.

[0066] 3.3.5.2 According to Calculate the i-th advantage value A i The calculation process is as follows:

[0067]

[0068] It is L R Calculate the average operation. It is L R The common difference operation involves standardizing the i-th reward value to obtain A. i .

[0069] 3.3.5.3 If i ≤ S, let i = i + 1, and then A i Store in A, then go to 3.3.5.2; if i>S, obtain A consisting of S dominant values, A={A1,…,A i ,…A S }, go to 3.3.5.4.

[0070] 3.3.5.4 The group advantage calculation module sends the advantage value list A to the strategy model update module.

[0071] 3.3.6 The strategy model update module receives the reference model input by the user and receives A from the group advantage calculation module. Let the weight parameters of the reference model be θ. old Let θ be the weight parameter of the policy model that needs to be updated. The policy model update module uses the GRPO reinforcement learning algorithm to encourage the policy model to generate answers with higher advantage in the group, while reducing the text probability distribution π of the current policy model's output. θ Text probability distribution deviating from the reference model output The degree of divergence (i.e., KL divergence value). Calculate the optimization loss value.

[0072] In formula (11), q represents the input query value, which consists of the image and prompt text entered by the user, o i This represents the i-th output text. The first term in formula (11) is min[d1·A] i ,d2·A i ] represents the weighted advantage value, i.e., the answer with higher advantage generated by the policy model in the group. d1 represents the difference between the output text of the current policy model and the frozen reference model. d2 represents the clipping of extreme advantage values ​​using clipping (truncation operation) to prevent training instability. ∈ in formula (12) is the truncation parameter, set to 0.2. The second term in formula (11) This involves penalizing the deviation of the current policy model's weight parameters from the reference model's weight parameters using KL divergence, in order to maintain training stability. β is a tradeoff between min[d1·A] i ,d2·A i ]and The contribution parameter is set to 0.01.

[0073] 3.3.7 The optimization loss value calculated using formula (11) Using the AdamW optimizer defined in Section 3.2, the weight parameters of the policy model are optimized through gradient backpropagation to obtain the Nth... b The policy model trained in batches;

[0074] 3.3.8 If N b ≤N T / B, let N b =N b +1, go to 3.3.2; if N b >N T / B, saves the trained policy model parameters and weights W. θ The trained policy model and the trained threat object reasoning and detection system based on the GRPO reinforcement learning algorithm were obtained.

[0075] The fourth step involves using the trained threat object inference and detection system based on the GRPO reinforcement learning algorithm to analyze the user-input image I captured by the car's front-facing camera. test To perform threat object detection and obtain the bounding box coordinates and threat level of the threat object, the method is as follows:

[0076] 4.1 The policy model after training receives the user-input image to be detected, I. test and prompt text I text (with I in 3.3.1) text (Same), the image to be detected I test and input text I text Multimodal processing is performed to obtain the image to be detected, I. test Single text output I′ output The strategy model in the image I to be detected test During detection, it is no longer necessary to sample multiple candidate output texts for model training; instead, only a single text output is generated.

[0077] 4.2 The text parsing module of the reward value calculation module receives I′ output Parse I′ output The algorithm includes image-level threat inference, object-level threat inference, object-level bounding box prediction, and object-level threat level prediction. The object-level bounding box prediction and object-level threat level prediction are then used as the final prediction results.

[0078] This invention proposes an innovative method based on a multimodal large language model. By employing the reinforcement learning algorithm GRPO (Group Relative Policy Optimization), this invention significantly improves the model's overall performance in threat understanding and reasoning, threat object detection, and threat level assessment.

[0079] The following beneficial effects can be achieved by using the present invention:

[0080] 1. This invention employs the GRPO reinforcement learning algorithm to train the policy model, significantly enhancing the model's understanding and reasoning ability regarding threatening objects in autonomous driving. By standardizing the model's output format, the model adaptively generates thought processes and responses, avoiding the manual design of thought chain prompts and achieving a higher level of intelligence.

[0081] 2. The reward value calculation module in the threat object detection system of this invention includes a text parsing module, an output format reward value calculation module, a non-repeating output reward value calculation module, a predicted quantity reward value calculation module, and an adder. This reward value calculation module requires no manual annotation and enhances the model's object-by-object reasoning ability through structured reasoning and responses. It not only achieves image-level reasoning analysis but also performs individual object-level reasoning analysis combined with context for each detected threat object. Furthermore, the non-repeating output reward value calculation module calculates a non-repeating output reward value for each output text in the output text set, avoiding duplicate outputs by the policy model, mitigating the "illusion" of the policy model, and allowing each detected object to be considered individually and fully. The predicted quantity reward value calculation module prevents single ill-conditioned solutions caused by non-repeating output reward values, achieving an effective trade-off between detection comprehensiveness and reasoning accuracy. Through the design of the above reward value calculation module, a multimodal large language model can enhance the perception and reasoning capabilities of threat objects in an unsupervised manner, thereby enhancing the safety of autonomous driving systems.

[0082] 3. The invention was evaluated using multiple autonomous driving datasets such as CODA, Waymo, Argoverse 2, KITTI, and nuScenes. The invention was compared with Qwen2.5-VL and the latest inference model VisionReasoner. The invention showed that the threat object detection accuracy and threat level estimation accuracy were significantly improved compared with the above two methods. Attached Figure Description

[0083] Figure 1 This is a training logic structure diagram of the threat object detection system based on reinforcement learning constructed in the first step of this invention.

[0084] Figure 2 This is a diagram showing the inference logic structure of the trained policy model in the reinforcement learning-based threat detection system constructed in the first step of this invention when performing threat detection on user-input prompt text and images.

[0085] Figure 3 This is the overall flowchart of the present invention.

[0086] Figure 4 This is a visual comparison chart of the present invention. Detailed Implementation

[0087] Specific examples of the present invention will be described below with reference to the accompanying drawings.

[0088] like Figure 3 As shown, the present invention includes the following steps:

[0089] The first step is to build a threat object reasoning and detection system based on reinforcement learning. For example... Figure 1 As shown, the threat object reasoning and detection system consists of a policy model, a reward value calculation module, a group advantage calculation module, and a policy model update module. The reward value calculation module, group advantage calculation module, and policy model update module are only used during policy model training.

[0090] The policy model is a multimodal large language model connected to the reward calculation module and the policy model update module. The policy model receives user input images and prompt text, and performs multimodal information processing on the input images and prompt text. The multimodal large language model needs to have multimodal processing capabilities and support bounding box detection, such as Qwen2.5VL-7B. The network weight parameters of the multimodal large language model use VisionReasoner. During policy model training, the policy model generates a batch of candidate output texts from the user input prompt text and images, obtaining an output text set, which is then sent to the reward calculation module. The policy model receives the optimized loss value from the policy model update module, updates the policy model weights based on the optimized loss value, and obtains the trained policy model. When the trained policy model performs threat object detection on the user input prompt text and images (e.g., ...), ... Figure 2 As shown, the policy model performs multimodal processing on the prompt text and image to obtain the output text. The text parsing module in the reward value calculation module receives the output text and parses it to obtain image-level threat inference content, object-level bounding box prediction, object-level threat level prediction, and object-level threat inference content. The object-level bounding box prediction and object-level threat level prediction (high threat, medium threat, low threat) are used as the final output.

[0091] The reward value calculation module is connected to the policy model and the group advantage calculation module. During policy model training, the reward value calculation module calculates the reward value for each text in the output text set received from the policy model, forming a final reward value list. The reward value calculation module consists of a text parsing module, an output format reward value calculation module, a non-repeating output reward value calculation module, a prediction quantity reward value calculation module, and an adder. The text parsing module parses the output text set, extracting the image-level threat inference content and object-level response content for each output text, resulting in a parsed output text set. This parsed output text set is then sent to the output format reward value calculation module, the non-repeating output reward value calculation module, and the prediction quantity reward value calculation module. The object-level response content consists of object-level bounding box prediction, object-level threat level prediction, and object-level threat inference content. The output format reward value calculation module calculates the output format reward value for each output text in the output text set; the non-repeating output reward value calculation module calculates the non-repeating output reward value for each output text in the output text set; and the prediction quantity reward value calculation module calculates the prediction quantity reward value for each output text in the output text set. The adder sums the three reward values ​​for each output text in the above output text set to obtain the final reward value for each output text in the output text set. These final reward values ​​are placed in the final reward value list and sent to the group advantage calculation module.

[0092] The group advantage calculation module is connected to the reward value calculation module and the strategy model update module. The group advantage calculation module receives the final reward value list from the reward value calculation module, calculates the mean and standard deviation of all final reward values ​​in the list, and standardizes the list to obtain the advantage performance of each final reward value relative to the group. This advantage performance is then placed into a relative group advantage value list, which is sent to the strategy model update module.

[0093] The policy model update module is connected to the group dominance calculation module and the policy model. The policy model update module receives a reference model input by the user and calculates an optimization loss value based on the list of relative group dominance values ​​received from the group dominance calculation module. This optimization loss value is used for gradient backpropagation to update the weight parameters of the policy model. The optimization loss value calculation includes the Kullback-Leibler (KL) divergence loss between the policy model and the reference model, and the group dominance loss. The reference model and the policy model use the same multimodal large language model, which is an untrained multimodal large language model with frozen weight parameters (i.e., the weights of the reference model will not change). The KL divergence loss is used to penalize the deviation between the updated policy model and the reference model, thereby maintaining training stability; the group dominance loss is used to encourage the generation of text output with high reward values, which better aligns with human preferences.

[0094] The second step is to construct the training and test sets, using the following method:

[0095] 2.1 We collect rare class images from autonomous driving scenarios as a threat object inference and detection dataset. The method is as follows:

[0096] 2.1.1 A subset of the validation set (1000 images) of the CODA autonomous driving scenario dataset is used as the training dataset D for the threat object inference and detection system. train Training dataset D train The number of images in the dataset is NT = 1000; 2.1.2 A subset of the test set of the autonomous driving scene dataset CODA (500 images), a subset of the autonomous driving dataset Waymo (122 images), a subset of the autonomous driving dataset Argoverse 2 (201 images), a subset of the autonomous driving dataset KITTI (69 images), and a subset of the autonomous driving dataset Nuscenes (44 images) are used as the threat object inference detection test dataset D. test D test It contains a total of M = 936 images.

[0097] 2.1.3 Find the driver in position G to position D test The M images in the dataset are manually annotated to obtain the manually annotated D. test Each manually labeled object includes the bounding box location of the threat object and its threat level (high threat, medium threat, low threat). G is a positive integer greater than 1.

[0098] 2.1.4 To further enhance the quality of annotation, the manually annotated D... test Samples with significant disagreements (labeled objects with threat level differences greater than one level) were reviewed, and a consensus was finally reached (labeled objects with threat level differences less than or equal to one level), resulting in the reviewed D. test , let it be DD test .

[0099] 2.2 For the training dataset D train Images and DD test The image is preprocessed using the following method:

[0100] 2.2.1 For the training dataset D train T images are scaled down to 840×840 pixels. The resulting preprocessed training dataset is denoted as D′. train .

[0101] 2.2.2 Regarding DD test M images were scaled down to 840×840 pixels. Meanwhile, DD... test The bounding box annotations in the dataset are scaled proportionally. This yields the preprocessed, reviewed test dataset, denoted as DD′. test .

[0102] The third step involves using the gradient backpropagation method to train the policy model on the reinforcement learning-based threat detection system constructed in the first step. This yields both the trained policy model and the trained reinforcement learning-based threat detection system. The method is as follows:

[0103] 3.1 Initialize the weight parameters of the policy model and reference model in the reinforcement learning-based threat object detection system. The weight parameters of the policy model and reference model are initialized using the weight parameters of the VisionReasoner model (an inference model retrained on the Qwen2.5VL-7B model).

[0104] 3.2 Setting Training Parameters for the Reinforcement Learning-Based Threat Detection System. AdamW was selected as the model training optimizer, with a learning rate of 0.000001 and a weight decay hyperparameter of 0.01. The batch size for network training was B = 16. The initial GRPO population sampling size was S = 8.

[0105] 3.3 The training of the threat object inference and detection system based on reinforcement learning is performed by calculating the final reward value list of the current output text set through the reward value calculation module, then calculating the relative group advantage value set through the group advantage calculation module, and finally calculating the optimized loss value and updating the policy model using the policy model update module until training is completed, resulting in the trained policy model. The specific method is as follows:

[0106] 3.3.1 Initialize batch sequence number N b =1. The strategy model reads the Nth bit of user input. b The prompt text in the batch is denoted as I. text I text The content includes the following prompts: Role prompt: "You are a professional driver with extensive driving experience."; Task prompt: "This is an image obtained from the vehicle's front camera. You need to detect the bounding boxes of threatening objects and estimate their threat level. Threat levels are categorized as high, medium, and low threat."; Output format prompt: "In..." <think> and< / think> The tag outputs image-level threat inference content, in <answer> and< / answer> The tag outputs object-level responses. The object-level responses are a JSON list. Each element contains a "think" key representing object-level threat inference, a "bbox_2d" key representing object-level bounding box prediction, and a "threat_level" key representing object-level threat level prediction. The number of elements in the list represents the number of predicted threat objects. The output format is: <think> Image-level threat reasoning content< / think> , <answer> [{"bbox_2d": object-level bounding box prediction, "think": object-level threat inference content, "threat_level": object-level threat level prediction},...]< / answer> ".

[0107] 3.3.2 The policy model is derived from the preprocessed training dataset D′ train Read the Nth b The batch contains B input images, which are represented as a matrix I of size B×H×W×3. train Where H represents the width of the input image, W represents the height of the input image, and "3" represents the three RGB channels of the image.

[0108] 3.3.3 Strategy Model for I train and I text Multimodal processing is performed to obtain the output text set I. output I output The output text has S group dominance sampling. Send to the reward value calculation module. The strategy model outputs according to the output format prompts required in 3.3.1, for example, outputting " <think> Image-level threat reasoning content< / think> , <answer> [{"bbox_2d": object-level bounding box prediction, "think": object-level threat inference content, "threat_level": object-level threat level prediction},...]< / answer> ".

[0109] 3.3.4 The reward value calculation module receives I output I is calculated using an unsupervised reward value calculation method. output The reward value corresponds to S output texts. The text parsing module parses the content of the output texts and outputs them as key-value pairs in JSON format, resulting in a parsed set of output texts. The output format reward value calculation module, the non-repeating output reward value calculation module, and the prediction quantity reward value calculation module simultaneously calculate the reward value for each of the S output texts. output The reward value is calculated from the S output texts, and then summed by an adder to obtain the final reward value, resulting in a list of S final reward values. This list is then sent to the group advantage calculation module. The specific method is as follows:

[0110] 3.3.4.1 The text parsing module receives I from the policy model output , to I output The text is parsed to obtain a set of parsed text. Will The method for sending the output format reward value calculation module is as follows:

[0111] 3.3.4.1.1 Initialize variable i = 1, and initialize the parsed text set. It is an empty set.

[0112] 3.3.4.1.2 Output the i-th text. Perform text parsing. Initialize. Parsed text output An empty JSON dictionary. Extract... middle <think> and< / think> The content between tags, as Image-level threat reasoning content. Extraction middle <answer> and< / answer> The content between tags, as The object-level response content is further parsed to obtain a JSON-formatted list, denoted as the object-level response list. Each list element consists of "bbox_2d" (object-level bounding box prediction), "think" (object-level threat inference content), and "threat_level" (object-level threat level prediction). The number of elements in the JSON-formatted list represents the number of predicted threat objects.

[0113] 3.3.4.1.3 If i ≤ S, let i = i + 1, then... deposit Go to 3.3.4.1.2; If i > S, then the parsed text set consists of S parsed texts. Go to 3.3.4.2.

[0114] 3.3.4.2 Output Format: The reward value calculation module receives data from the text parsing module. right The output format reward value is calculated for each parsed output text, using the following method:

[0115] 3.3.4.2.1 Initialize variable i = 1, initialize the output format reward value list It is an empty list.

[0116] 3.3.4.2.2 Output the i-th parsed text Computational thinking format reward value: If Correct match <think>and <answer>Tags, i.e. If neither the image-level threat reasoning nor the object-level response is empty, then let Mindset Reward Value Otherwise

[0117] 3.3.4.2.3 Output the i-th parsed text Calculate reward based on answer format. Extract the list of object-level answers. Calculate. Answer format reward

[0118]

[0119] in, This is the first indicator function. It returns 1.0 if "bbox_2d" can be effectively parsed from all elements in the object-level answer list, and 0.0 otherwise. This is the second indicator function. It returns 1.0 if "think" can be effectively parsed from all elements in the object-level answer list, and 0.0 otherwise. This is the third indicator function. It returns 1.0 if "think" can be effectively parsed from all elements in the object-level answer list, and 0.0 otherwise.

[0120] 3.3.4.2.4 Reward Values ​​for Thinking Format Rewards for answer format Add them together to get Output format reward value

[0121]

[0122] 3.3.4.2.5 If i ≤ S, let i = i + 1, then... deposit Go to 3.3.4.2.2; if i > S, then the result consists of S output format reward values. Will Send to the adder, then proceed to 3.3.4.3.

[0123] 3.3.4.3 The non-repeating output reward value calculation module receives from the text parsing module. right Each parsed output text in the model receives a non-repeating output reward value. Repetitive outputs often indicate a lack of thoughtful consideration in the model and may even produce illusory content. To address this issue, this invention introduces a non-repeating output reward, penalizing repeated inference text content and repeated object box outputs, thereby achieving more context-aware and differentiated analysis. The process of calculating the non-repeating output reward value is as follows:

[0124] 3.3.4.3.1 Initialize i = 1, and initialize a list of non-repeating reward values. It is an empty list.

[0125] 3.3.4.3.2 Calculate the non-repetitive reward value for the i-th image-level thought. from Extract image-level threat inference content from the text, and split the text into sentences to obtain the set T of clauses for the i-th image-level threat inference content. think , There are E clauses in total. Calculate the clause set T. think Non-repeating reward value R think-nr (T think The process is as follows:

[0126] Indicates traversal of T think All clause pairs in T (a clause pair refers to T) think Each element in the text is paired with another element in the text. Clauses and If the clauses are identical, increment the count by 1. If the final count is less than 2, then let... Image-level thinking non-repetitive reward value Otherwise

[0127] 3.3.4.3.3 Calculate the non-repeating reward value for the i-th bounding box. from Retrieve the object-level response list, then extract the content of the "bbox_2d" key from the object-level response list to obtain the bounding box set T of the bounding box prediction for the i-th object level. bbox , T bbox There are K object elements in total. Let T be the bounding box of the k-th predicted threat object. Calculate the bounding box set T. bbox Non-repeating reward value R bbox-nr (T bbox The process is as follows:

[0128]

[0129] |T bbox | represents T bbox The number of elements in the set, |unique(T) bbox )| represents T bbox The number of distinct elements in a set. If |T bbox |==|unique(T bbox )|, which means T bbox If all bounding boxes in the set are distinct, then let Non-repeating reward value of the object frame Otherwise

[0130] 3.3.4.3.4 Calculate the non-repeating reward value for the i-th object-level thinking. from Extract the object-level response list, then extract the "think" content from the object-level response list to obtain the threat reasoning content set T for the i-th object level. obj-think , There are K elements in total. Let T be the object-level inference content for the k-th predicted threat object. Calculate the threat inference content set T. obj-think Non-repeating reward value R objthink-nr (T obj-think The process is as follows:

[0131] Indicates traversal of T obj-think All object-level reasoning content text pairs (T) obj-think Each element in the text is paired with another element in the text. and If they are exactly the same, increment the count by 1. If the final count is less than 2, then let... Non-repetitive reward value for object-level thinking Otherwise

[0132] 3.3.4.3.5 Non-repetitive reward value for image-level thinking Non-repeating reward value of the object frame Object-level thinking non-repetitive reward value Add them together to get the i-th non-repeating output reward value.

[0133]

[0134] 3.3.4.3.6 If i ≤ S, let i = i + 1, then... deposit In section 3.3.4.3.2; if i > S, obtain S non-repeating output reward values. Will Send to the adder, then proceed to 3.3.4.4.

[0135] 3.3.4.4 The predicted quantity reward value calculation module receives data from the text parsing module. While rewarding the avoidance of duplicate outputs can improve the independence of object-level reasoning, it may also lead the model to output only one prediction box to avoid repetition, resulting in missed detections. Therefore, this invention introduces a prediction quantity reward to encourage the model to make multiple reasonable predictions. However, too many predictions may lead to excessively long outputs and affect the reasoning quality for each object. To balance this issue, this invention applies a flexible constraint on the number of predictions, incentivizing the policy model to improve prediction coverage within a reasonable range while maintaining the accuracy of reasoning for individual objects. The process of calculating the predicted quantity reward value is as follows:

[0136] 3.3.4.4.1 Initialize i = 1, initialize the prediction quantity reward value list An empty list.

[0137] 3.3.4.4.2 Calculate the reward value for the i-th predicted quantity. from Extract the object-level response list, then extract the "think" content from the object-level response list to obtain the threat reasoning content set T for the i-th object level. obj-think , There are K elements in total. This is the object-level inference content for the k-th predicted threat object. Since the prediction quantity reward only requires the number of predicted object boxes K to be calculated, the prediction quantity reward value R is calculated based on the number of predicted object boxes K. num The process for (K) is as follows:

[0138] N is the constraint inflection point parameter, set to 5. Then the reward value for the i-th predicted quantity...

[0139] 3.3.4.4.3 If i ≤ S, let i = i + 1, then... deposit In section 3.3.4.4.2; if i > S, obtain S non-repeating output reward values. Will Send to the adder, then proceed to 3.3.4.5.

[0140] 3.3.4.5 Adder Receiver Output Format: List of Reward Values Output a list of reward values ​​without duplicates. List of predicted quantity reward values Summing each of the three lists element by element yields a list of S total reward values, denoted as the final reward value list L. R , This is the i-th final reward value. Let L... R Send to the group advantage calculation module.

[0141] 3.3.5 The group advantage calculation module receives L from the reward value calculation module. R According to L R calculate The advantage value, obtained The corresponding advantage values ​​A1,…,A i ,…A S Send the advantage value list A to the policy model update module, where A = {A1, ..., A...} i ,…A S The method is:

[0142] 3.3.5.1 Initialize i = 1, and initialize the advantage value list A as an empty list.

[0143] 3.3.5.2 According to Calculate the i-th advantage value A i The calculation process is as follows:

[0144]

[0145] It is L R Calculate the average operation. It is L R The common difference operation involves standardizing the i-th reward value to obtain A. i .

[0146] 3.3.5.3 If i ≤ S, let i = i + 1, and then A i Store in A, then go to 3.3.5.2; if i>S, obtain A consisting of S dominant values, A={A1,…,A i ,…A S }, go to 3.3.5.4.

[0147] 3.3.5.4 The group advantage calculation module sends the advantage value list A to the strategy model update module.

[0148] 3.3.6 The strategy model update module receives the reference model input by the user and receives A from the group advantage calculation module. Let the weight parameters of the reference model be θ. old Let θ be the weight parameter of the policy model that needs to be updated. The policy model update module uses the GRPO reinforcement learning algorithm to encourage the policy model to generate answers with higher advantage in the group, while reducing the text probability distribution π of the current policy model's output. θ Text probability distribution deviating from the reference model output The degree of divergence (i.e., KL divergence value). Calculate the optimization loss value.

[0149] In formula (11), q represents the input query value, which consists of the image and prompt text entered by the user, o i This represents the i-th output text. The first term in formula (11) is min[d1·A] i ,d2·A i ] represents the weighted advantage value, i.e., the answer with higher advantage generated by the policy model in the group. d1 represents the difference between the output text of the current policy model and the frozen reference model. d2 represents the clipping of extreme advantage values ​​using clipping (truncation operation) to prevent training instability. ∈ in formula (12) is the truncation parameter, set to 0.2. The second term in formula (11) This involves penalizing the deviation of the current policy model's weight parameters from the reference model's weight parameters using KL divergence, in order to maintain training stability. β is a tradeoff between min[d1·A] i ,d2·A i ]and The contribution parameter is set to 0.01.

[0150] 3.3.7 The optimization loss value calculated using formula (11) Using the AdamW optimizer defined in Section 3.2, the weight parameters of the policy model are optimized through gradient backpropagation to obtain the Nth... b The policy model trained in batches;

[0151] 3.3.8 If N b ≤N T / B, let N b =N b +1, go to 3.3.2; if N b >N T / B, saves the trained policy model parameters and weights W. θ The trained policy model and the trained threat object reasoning and detection system based on the GRPO reinforcement learning algorithm were obtained.

[0152] The fourth step involves using the trained threat object inference and detection system based on the GRPO reinforcement learning algorithm to analyze the user-input image I captured by the car's front-facing camera. test To perform threat object detection and obtain the bounding box coordinates and threat level of the threat object, the method is as follows:

[0153] 4.1 The policy model after training receives the user-input image to be detected, I. test and prompt text I test (with I in 3.3.1) text (Same), the image to be detected I test and input text I test Multimodal processing is performed to obtain the image to be detected, I. test Single text output I′ output The strategy model in the image I to be detected test During detection, it is no longer necessary to sample multiple candidate output texts for model training; instead, only a single text output is generated.

[0154] 4.2 The text parsing module of the reward value calculation module receives I′ output Parse I′ output The algorithm includes image-level threat inference, object-level threat inference, object-level bounding box prediction, and object-level threat level prediction. The object-level bounding box prediction and object-level threat level prediction are then used as the final prediction results.

[0155] To verify the effectiveness of this invention, the preprocessed and reviewed test dataset DD′ constructed in step 2.1 for threat object reasoning detection was used. text That is, using DD′ text The images in the image were used as the detection objects to test the performance of the present invention. To evaluate the performance of the present invention in both threat object detection and threat level estimation, two types of evaluation metrics were used to measure performance.

[0156] This experiment used Ubuntu 20.04 and trained on eight NVIDIA L40 GPUs. The preprocessed and reviewed test dataset DD′ was used. test The driver is labeled with G, where G=5. The multimodal large language model of this invention is trained using the pre-trained weight parameters of the Qwen2.5-VL-7B and VisionReasoner-7B models.

[0157] In threat object detection tasks, although Average Precision (AP) is a commonly used metric for evaluating detection accuracy, it is not suitable for this task. This task classifies targets according to threat level, rather than mutually exclusive semantic categories. For example, if the model predicts an object bounding box that matches the ground truth bounding box in location, but the predicted threat level is "medium" while the ground truth label is "high," the traditional AP metric would consider it a complete error. However, in threat object detection tasks, such predictions, while deviating from the ground truth level, still demonstrate the ability to identify threat-related objects and should therefore receive a smaller penalty. To more reasonably evaluate the detection performance of such tasks, this invention proposes Conditional Recall (CRecall). CRecall alleviates the "black and white" penalty of the AP metric by redefining the evaluation objective: to recall as many real threat objects as possible, provided that the number of predicted bounding boxes does not exceed the number of ground truth labels. The formula for calculating CRecall is as follows:

[0158]

[0159] in, and They represent DD′ respectively test The bounding boxes at the labeled level and the bounding boxes at the object level are predicted. τ represents the IoU (Intersection over Union) threshold, which is set to 0.5.

[0160] In threat level estimation tasks, the model is also required to predict the threat level of each object. Since incorrectly classifying "high threat" as "medium" or "low threat" should be penalized differently, threat level prediction is closer to a regression task than a traditional classification task. Therefore, this invention uses L1 distance and L2 distance as evaluation metrics to measure the deviation between the model output and the true threat level. Compared to L1 distance, L2 distance incurs a greater penalty for larger prediction errors.

[0161] Based on the experimental results of this invention, the performance comparison of this invention in threat object detection tasks is shown in Table 1. The first column of Table 1 represents the methods, including Qwen2.5VL, VisionReasoner, and the method of this invention. Since the LISA model described in the background art cannot generate threat level predictions and Seg-zero cannot detect multi-target threat objects, it was not included in this experimental comparison. In the first row of Table 1, 1 (%), 2 (%), and 3 (%) represent the CRecall (Mean CRecall) index for low, medium, and high threat levels, respectively, and Mean (%) represents the average CRecall for the three threat levels. The upward arrow indicates that a higher C-Recall index is better. This invention significantly improves recall accuracy, achieving significant improvements of 31.2% and 9.7% in average C-Recall index compared to Qwen2.5VL and VisionReasoner, respectively. These results further emphasize the importance of introducing threat-related inference mechanisms in threat object detection tasks.

[0162] Table 1

[0163]

[0164] The performance comparison of this invention in threat level assessment is shown in Table 2. The first column of Table 1 represents the methods, including Qwen2.5VL, VisionReasoner, and the method of this invention. The first row of Table 2 represents the evaluation metrics, using L1 and L2 distances. The second row of Table 2 shows the L1 distances for the three threat levels (3 (high threat), 2 (medium threat), and 1 (low threat), and the L1 and L2 distances for the Mean (the average of the three threat levels). Downward arrows indicate that smaller L1 and L2 values ​​are better. Compared to Qwen2.5VL, this invention reduces the average L1 distance (Mean under L1 Distance) error by 0.32 and the average L2 distance error (Mean under L2 Distance) error by 0.99; compared to VisionReasoner, it reduces the average L1 and L2 distance errors by 0.17 and 0.55, respectively. These results further verify the effectiveness of this invention in threat level judgment, indicating that introducing a self-supervised reinforcement learning mechanism is crucial for improving the model's inference ability.

[0165] Table 2

[0166]

[0167] The invention is visualized and analyzed. For example... Figure 4 As shown, Figure 4 (a) is DD' test A manually labeled document Figure 4 (b) shows the test results for Qwen2.5VL. Figure 4 (c) shows the detection results of VisionReasoner and Figure 4 (d) is a visual analysis of the detection results of this invention, with red boxes representing high-threat objects, yellow boxes representing medium-threat objects, and blue boxes representing low-threat objects. The detection results of this invention ( Figure 4 (d) Comparison of Qwen2.5VL detection results ( Figure 4 (b) and VisionReasoner detection results ( Figure 4 (c) This method can detect more threatening objects, including "pedestrians on the roadside nearby", "pedestrians in the middle of the road in the distance", and "motorcycles and cars parked on the roadside", and can give more reasonable threat levels. For example, "the aerial work platform directly opposite" is given a reasonable high threat rating. These visualization results demonstrate that the method of the present invention has significant advantages in both the detection coverage and accuracy of threatening objects, and also estimates the threat level more reasonably.< / answer> < / think> < / answer> < / think> < / think>

Claims

1. A threat object detection method based on reinforcement learning, characterized in that... Includes the following steps: The first step is to construct a threat object reasoning and detection system based on reinforcement learning. The threat object reasoning and detection system consists of a policy model, a reward value calculation module, a group dominance calculation module, and a policy model update module. The reward value calculation module, group dominance calculation module, and policy model update module are only used during policy model training. The reward value calculation module consists of a text parsing module, an output format reward value calculation module, a non-repeating output reward value calculation module, a predicted quantity reward value calculation module, and an adder. The strategy model is connected to the reward calculation module and the strategy model update module. The strategy model receives user-input images and prompt text, and performs multimodal information processing on the input images and prompt text. During strategy model training, the strategy model generates a batch of candidate output texts from the user-input prompt text and images, obtaining an output text set, which is then sent to the reward calculation module. The strategy model receives the optimized loss value from the strategy model update module to update the strategy model's weights, resulting in the trained strategy model. When the trained strategy model performs threat object detection on the user-input prompt text and images, it performs multimodal processing on the prompt text and images, obtaining output text. The text parsing module in the reward calculation module receives the output text and parses it to obtain image-level threat inference content, object-level bounding box prediction, object-level threat level prediction, and object-level threat inference content. The object-level bounding box prediction and object-level threat level prediction are used as the final output; the threat level includes high threat, medium threat, and low threat. The reward value calculation module calculates the reward value for the set of output texts received from the strategy model, and obtains the reward value for each text in the set of output texts, forming a final reward value list. The group advantage calculation module calculates the mean and standard deviation of all final reward values ​​in the final reward value list, and standardizes the final reward value list to obtain the advantage performance of each final reward value relative to the group. The strategy model update module receives the reference model input by the user, calculates the optimization loss value for the relative group advantage value list, and uses it for gradient backpropagation to update the weight parameters of the strategy model. The second step is to construct the training and test sets, using the following method: Step 2.1 Collect rare class images from autonomous driving scenarios as a threat object inference and detection dataset to obtain the training dataset for the threat object inference and detection system. Threat Object Reasoning Detection Test Dataset ;right Manual annotation and review were conducted to obtain the reviewed version. , order ; Step 2.2 Process the training dataset Images and The images are preprocessed to obtain the preprocessed training dataset. and the preprocessed and reviewed test dataset ; The third step involves training the policy model using the gradient backpropagation method with a reinforcement learning-based threat detection system. This yields both the trained policy model and the trained reinforcement learning-based threat detection system. The method is as follows: Step 3.1 Initialize the weight parameters of the policy model and the reference model in the reinforcement learning-based threat object detection system; initialize the weight parameters of the policy model and the reference model using the weight parameters of the VisionReasoner model; Step 3.2 Set the training parameters for the reinforcement learning-based threat detection system; select AdamW as the model training optimizer, and set the batch size for network training. The initial population sampling size S of GRPO is a positive integer; Step 3.3 Train the threat object inference and detection system based on reinforcement learning. This involves calculating the final reward value list for the current output text set using the reward value calculation module, then calculating the relative group advantage value set using the group advantage calculation module, and finally calculating the optimized loss value and updating the policy model using the policy model update module. This process continues until training is complete, resulting in the trained policy model. The specific method is as follows: Step 3.3.1 Initialize batch serial number =1; The strategy model reads the first... Prompt text in batch ; The content includes character prompts, task prompts, and output format prompts; Step 3.3.2 The policy model is derived from the preprocessed training dataset. Read the first The input images in the batch total This image will The image is denoted as having a size of matrix form ; where H represents the width of the input image, W represents the height of the input image, and "3" represents the three RGB channels of the image; Step 3.3.3 Strategy Model Pair and Perform multimodal processing to obtain the output text set. , The output text has S group dominance sampling; Send to the reward value calculation module; the strategy model outputs according to the format required by the output format prompt; Step 3.3.4 Reward Value Calculation Module Receives The unsupervised reward value calculation method is used to calculate... The reward values ​​correspond to S output texts; the text parsing module parses the content of the output texts and outputs them as key-value pairs in JSON format, resulting in a parsed set of output texts. The content includes image-level threat inference content and object-level response content for each output text. The object-level response content consists of object-level bounding box prediction, object-level threat level prediction, and object-level threat inference content. The output format reward value calculation module, the non-repeating output reward value calculation module, and the prediction quantity reward value calculation module simultaneously calculate the reward value for each output text. The reward values ​​for the S output texts are calculated, and then summed by an adder to obtain the final reward value, resulting in a list of S final reward values. , =[ ,…, ,…, ], It is the i-th final reward value; Send to the group dominance calculation module; Step 3.3.5 The group advantage calculation module receives the reward value from the reward value calculation module. ,according to calculate The advantage value, obtained Corresponding advantage value List the advantages Send to the strategy model update module. ; Step 3.3.6 The strategy model update module receives the reference model input by the user and receives it from the group advantage calculation module. The GRPO reinforcement learning algorithm is used to encourage the policy model to generate answers with higher advantage in the group, while reducing the probability distribution of the text output by the current policy model. Text probability distribution deviating from the reference model output The degree of divergence is the KL divergence value. The weight parameters are for the reference model. Set the weight parameters for the policy model that needs updating; calculate the optimization loss value. : ，（11）； ，（12）； In formula (11) The input query value consists of an image entered by the user and a prompt text. This represents the i-th output text; This represents the weighted advantage value, corresponding to the answer that the policy model generates with a higher advantage in the group. This indicates the difference between the output text of the current policy model and the frozen reference model. This indicates that the clip operation is used to prune extreme dominance values, thereby preventing training instability; in formula (12) It is the truncation parameter; in formula (11) This involves using KL divergence to penalize the degree to which the weight parameters of the current policy model deviate from the weight parameters of the reference model, in order to maintain training stability; It is a trade-off and Parameters of contribution; Step 3.3.7 adopts The AdamW optimizer is used to optimize the weight parameters of the policy model through gradient backpropagation, resulting in the... The policy model trained in batches; Step 3.3.8 If , make Proceed to step 3.3.2; if Save the parameter weights of the trained policy model. The trained policy model and the trained threat object reasoning and detection system based on the GRPO reinforcement learning algorithm were obtained. The fourth step involves using the trained threat object inference and detection system based on the GRPO reinforcement learning algorithm to analyze the user-input images captured by the car's front-facing camera. To perform threat object detection and obtain the bounding box coordinates and threat level of the threat object, the method is as follows: Step 4.1 The trained policy model receives the image to be detected input by the user. and prompt text The image to be detected and input text Perform multimodal processing to obtain Text output ; Step 4.2 The text parsing module of the reward value calculation module receives... Analyze The system includes image-level threat inference content, object-level threat inference content, object-level bounding box prediction, and object-level threat level prediction; and uses the object-level bounding box prediction and object-level threat level prediction as the final prediction results.

2. The threat object detection method based on reinforcement learning as described in claim 1, characterized in that... The policy model is a multimodal large language model, which has multimodal processing capabilities and supports bounding box detection. The reference model is the same multimodal large language model used by the policy model, which is an untrained multimodal large language model with frozen weight parameters.

3. The threat object detection method based on reinforcement learning as described in claim 2, characterized in that... The multimodal large language model is Qwen2.5VL-7B, and the network weight parameters of the multimodal large language model adopt VisionReasoner.

4. The threat object detection method based on reinforcement learning as described in claim 1, characterized in that... The reward value calculation module is connected to the strategy model and the group advantage calculation module. When training the strategy model, the text parsing module of the reward value calculation module parses the output text set, extracts the image-level threat reasoning content and object-level response content of each output text in the output text set, and obtains the parsed output text set. The parsed output text set is sent to the output format reward value calculation module, the non-repeating output reward value calculation module, and the prediction quantity reward value calculation module. The output format reward value calculation module calculates the output format reward value for each output text in the output text set, the non-repeating output reward value calculation module calculates the non-repeating output reward value for each output text in the output text set, and the prediction quantity reward value calculation module calculates the prediction quantity reward value for each output text in the output text set. The adder sums the three reward values ​​of each output text in the above output text set to obtain the final reward value of each output text in the output text set. These final reward values ​​are put into the final reward value list and sent to the group advantage calculation module. The group advantage calculation module is connected to the reward value calculation module and the strategy model update module. The group advantage calculation module receives the final reward value list from the reward value calculation module, puts the calculated advantage performance into the relative group advantage value list, and sends the relative group advantage value list to the strategy model update module. The policy model update module is connected to the group dominance calculation module and the policy model. The optimization loss calculation of the policy model update module includes the KL divergence loss between the policy model and the reference model and the group dominance loss. The KL divergence loss is used to penalize the deviation between the updated policy model and the reference model, thereby maintaining training stability. The group dominance loss is used to encourage the generation of text output with high reward values, which can better align with human preferences.

5. The threat object detection method based on reinforcement learning as described in claim 1, characterized in that... The method described in step 2.1 for collecting rare class images of autonomous driving scenarios as a threat object inference and detection dataset is as follows: Step 2.1.1 Use a subset of the validation set of the CODA autonomous driving scenario dataset as the training dataset for the threat object inference and detection system. The CODA validation set subset contains 1000 images, while the training dataset... Number of images in 1000; Step 2.1.2 uses a subset of the test set of the autonomous driving scenario dataset CODA (500 images), the subset of the autonomous driving dataset Waymo (122 images), the subset of the autonomous driving dataset Argoverse 2 (201 images), the subset of the autonomous driving dataset KITTI (69 images), and the subset of the autonomous driving dataset Nuscenes (44 images) as a test dataset for threat object inference detection. ; Total of 936 images; Step 2.1.3 Find the driver in position G. The M images in the dataset are manually annotated to obtain the manually annotated images. Each manually labeled document includes the bounding box location of the threatening object and the threat level of the threatening object; G is a positive integer greater than 1. Step 2.1.4: After manual annotation Samples with significant disagreements, i.e., labeled objects with threat level differences greater than one level, were reviewed. A consensus was ultimately reached that labeled objects with threat level differences of one level or less were eligible for the review. , order .

6. The threat object detection method based on reinforcement learning as described in claim 1, characterized in that... Step 2.2 describes the process of training the dataset. Images and The method for image preprocessing is: Step 2.2.1 Process the training dataset The T images were scaled down to a size of 1. ; The preprocessed training dataset is obtained, denoted as ; Step 2.2.2 For The M images are scaled up to a size of 1. At the same time, The bounding box annotations in the text are scaled proportionally. The preprocessed and reviewed test dataset is denoted as . .

7. The threat object detection method based on reinforcement learning as described in claim 1, characterized in that... Step 3.3.1 The prompts include: Character prompt: "You are a professional driver with extensive driving experience;"; Task prompt: "This is an image obtained from the vehicle's front camera. You need to detect the bounding boxes of threatening objects and estimate their threat level; threat levels are categorized as high, medium, and low threat"; Output format prompt: "In..." <think> and< / think> The tag outputs image-level threat inference content, in <answer> and< / answer> The tag outputs object-level responses; these responses are JSON-formatted lists. Each list element contains a "think" key representing object-level threat inference, a "bbox_2d" key representing object-level bounding box prediction, and a "threat_level" key representing object-level threat level prediction. The number of elements in the list represents the number of predicted threat objects. The output format is: " <think> Image-level threat reasoning content< / think> , <answer>[{"bbox_2d": object-level bounding box prediction, "think": object-level threat inference content, "threat_level": object-level threat level prediction}, …]< / answer> ".

8. The threat object detection method based on reinforcement learning as described in claim 1, characterized in that... The reward value calculation module described in step 3.3.4 receives... The unsupervised reward value calculation method is used to calculate... The method for obtaining a final reward value list consisting of S final reward values ​​corresponding to S output texts and sending the final reward value list to the group advantage calculation module is as follows: Step 3.3.4.1 The text parsing module receives data from the policy model. ,right The text is parsed to obtain a set of parsed text. ,Will The method for sending the output format reward value calculation module is as follows: Step 3.3.4.1.1 Initialize variables Initialize the parsed text set It is an empty set; Step 3.3.4.1.2 Output the i-th text. Perform text parsing; initialize Parsed text output An empty JSON dictionary; extract middle <think> and< / think> The content between tags, as Image-level threat reasoning content; extraction middle <answer>and< / answer > The content between tags, as The object-level response content is further parsed to obtain a JSON-formatted list, denoted as the object-level response list. Each list element consists of "bbox_2d" (object-level bounding box prediction), "think" (object-level threat inference content), and "threat_level" (object-level threat level prediction). The number of elements in the JSON-formatted list represents the number of predicted threat objects.< / answer> Step 3.3.4.1.3 If ,make ,Will deposit Proceed to step 3.3.4.1.2; if This yields a set of parsed text consisting of S parsed texts. , Proceed to step 3.3.4.2; Step 3.3.4.2 Output Format: The reward value calculation module receives the data from the text parsing module. ;right The output format reward value is calculated for each parsed output text, using the following method: Step 3.3.4.2.1 Initialize variables Initialize the output format to include a list of reward values. An empty list; Step 3.3.4.2.2 Output the i-th parsed text. Computational thinking format reward value: If Correct match <think>and <answer>Tags, i.e. If neither the image-level threat reasoning nor the object-level response is empty, then let Mindset Reward Value ;otherwise ;< / answer> < / think> Step 3.3.4.2.3 Output the i-th parsed text. Calculate the reward based on the answer format; from Extract the object-level answer list from the data; calculate... Answer format reward : ，（1）； in, This is the first indicator function. It returns 1.0 if "bbox_2d" can be effectively parsed from all elements in the object-level answer list, and 0.0 otherwise. This is the second indicator function. It returns 1.0 if "think" can be effectively parsed from all elements in the object-level answer list, and 0.0 otherwise. This is the third indicator function. It returns 1.0 if "think" can be effectively parsed from all elements in the object-level answer list, and 0.0 otherwise. Step 3.3.4.2.4 Calculate the reward value for the mindset format. Rewards for answer format Add them together to get Output format: reward value : ，（2）； Step 3.3.4.2.5 If ,make ,Will deposit Proceed to step 3.3.4.2.2; If This yields a reward value consisting of S output formats. , ,Will Send to the adder, proceed to step 3.3.4.3; Step 3.3.4.3 The non-repeating output reward value calculation module receives from the text parsing module. ;right For each parsed output text, a unique reward value is calculated and output; The process of calculating the non-repeating output reward value is as follows: Step 3.3.4.3.1 Initialization Initialize a list of non-repeating reward values. An empty list; Step 3.3.4.3.2 For Calculate the non-repetitive reward value for the i-th image-level thought. ;from Extract image-level threat inference content from the text, and split the text into sentences to obtain the set of clauses for the i-th image-level threat inference content. , There are E clauses in total; calculate the set of clauses. Non-repeating reward value The process is as follows: ，（4）； Indicates traversal All clause pairs (clause pair refers to...) Each element in the text is paired with another element in the text. Clauses and If the clauses are exactly the same, increment the count by 1; if the final count is less than 2, then let... Image-level thinking non-repetitive reward value Otherwise ; Step 3.3.4.3.3 For Calculate the non-repeating reward value for the i-th bounding box. ;from Retrieve the object-level response list, then extract the content of the "bbox_2d" key from the object-level response list to obtain the bounding box set predicted for the i-th object-level bounding box. , , There are K object elements in total. For the k-th predicted threat object, calculate the bounding box set. Non-repeating reward value The process is as follows: ，（5）； express The number of elements in the set. express The number of distinct elements in a set; if That is to say If all bounding boxes in the set are distinct, then let Non-repeating reward value of the object frame Otherwise ; Step 3.3.4.3.4 Calculate the non-repeating reward value for the i-th object-level thinking. ;from Extract the object-level answer list, then extract the "think" content from the object-level answer list to obtain the threat reasoning content set for the i-th object level. , There are K elements in total. For the object-level inference content of the k-th predicted threat object; calculate the threat inference content set. Non-repeating reward value The process is as follows: ，（6）； Indicates traversal All object-level reasoning content text pairs in the text. Each element in the text is paired with another element. and If they are exactly the same, increment the count by 1. If the final count is less than 2, then let... Non-repetitive reward value for object-level thinking Otherwise ; Step 3.3.4.3.5 Non-repetitive reward value for image-level thinking Non-repeating reward value for the bounding box Non-repetitive reward value for object-level thinking Add them together to get the i-th non-repeating output reward value. : ，（7）； Step 3.3.4.3.6 If ,make ,Will deposit In the middle, proceed to step 3.3.4.3.2; if This results in S unique output reward values. , ,Will Send to the adder, proceed to step 3.3.4.4; Step 3.3.4.4 The predicted quantity reward value calculation module receives the data from the text parsing module. right ,..., ,..., The process for calculating the reward value for the predicted quantity is as follows: Step 3.3.4.4.1 Initialization Initialize the list of predicted quantity reward values. An empty list; Step 3.3.4.4.2 For Calculate the reward value for the i-th predicted quantity. ;from Extract the object-level answer list, then extract the "think" content from the object-level answer list to obtain the threat reasoning content set for the i-th object level. , There are K elements in total. For the object-level inference content of the k-th predicted threat object; calculate the prediction quantity reward value for the number of predicted object boxes K. The process is as follows: ，（8）； N is the constraint inflection point parameter, set to 5; then the reward value for the i-th predicted quantity... = ; Step 3.3.4.4.3 If ,make ,Will deposit In the middle, proceed to step 3.3.4.4.2; if This results in S unique output reward values. , ,Will Proceed to step 3.3.4.5; Step 3.3.4.5 Adder receives and outputs a list of reward values ​​in the specified format. , Output a list of reward values ​​without duplicates List of predicted quantity reward values Summing each of the three lists element by element yields a list of S total reward values, denoted as the final reward value list. , =[ , …, ,…, ], It is the i-th final reward value; Send to the group advantage calculation module.

9. The threat object detection method based on reinforcement learning as described in claim 1, characterized in that... Step 3.3.5: Group Advantage Calculation Module calculate The advantage value is used to obtain a list of advantage values. The method to send the update to the policy model module is: Step 3.3.5.1 Initialization Initialize the list of advantage values An empty list; Step 3.3.5.2 According to Calculate the i-th advantage value The calculation process is as follows: ，（10）； Yes Calculate the average operation. Yes The common difference operation involves standardizing the i-th reward value to obtain... ; Step 3.3.5.3 If ,make ,Will deposit In the middle, proceed to step 3.3.5.2; if S advantage values ​​were obtained. , Proceed to step 3.3.5.4; Step 3.3.5.4 The group dominance calculation module will list the dominance values. Send to the strategy model update module.

10. The threat object detection method based on reinforcement learning as described in claim 1, characterized in that... In step 3.2, the learning rate of AdamW is set to 0.000001, and the hyperparameter "weight decay" of AdamW is set to 0.01; the batch size for network training is... =16; Initialize the GRPO population sampling number S=8.

11. The threat object detection method based on reinforcement learning as described in claim 1, characterized in that... The cutoff parameter in formula (12) described in step 3.3.6 Set to 0.2; parameter Set to 0.01.

Images