A target detection method incorporating reinforcement learning with human feedback

Abstract

This invention discloses a target detection method incorporating reinforcement learning with human feedback, comprising the following steps: collecting scene image data, using a general pre-trained target detection model for prediction, obtaining a set of prediction results; for each prediction, performing manual annotation to obtain a feedback dataset containing the category labels of the detection boxes; training a reinforcement learning model using the feedback dataset; using the target detection model to predict unlabeled scene data images, and using the reinforcement learning model to extract good targets from the prediction results, fine-tuning the detection model, using gradient descent algorithm, and updating parameters using a loss function to minimize the loss; repeating the previous steps to finally train a detection model specific to the target scene. This method uses a general pre-trained model to reduce training time and uses reinforcement learning based on human feedback to reduce the number of annotations, thereby improving target detection efficiency.

Description

Technical Field

[0001] This invention relates to the field of object detection, and more specifically to an object detection method that incorporates reinforcement learning based on human feedback. Background Technology

[0002] With the widespread use of surveillance videos, the volume of video data has surged. Analyzing these massive amounts of video is becoming increasingly important, and object detection is a key method for video analysis. Publicly available object detection models (such as the GroundingDINO model) are trained on open-source datasets. The real key to applying object detection is its ability to automatically identify the targets of interest to users in various surveillance scenarios. Traditional methods require retraining object detection models from scratch, typically necessitating extensive manual image annotation, which is extremely costly. Furthermore, the targets of interest differ across surveillance scenarios, requiring the re-annotation of large amounts of data for new scenarios, further increasing costs. Therefore, a major challenge is how to quickly adapt to new scenarios without requiring extensive new annotations. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this invention proposes a target detection method incorporating reinforcement learning based on human feedback. This method uses reinforcement learning based on human feedback to fine-tune the target detector, reducing training time, decreasing the number of annotations, and effectively improving target detection efficiency. The technical solution is as follows:

[0004] A target detection method incorporating reinforcement learning with human feedback includes the following steps:

[0005] Step 1: Collect n on-site image data, denoted as x. i , i∈[1,2,3,…,n], use a general pre-trained object detection model D to make predictions and obtain a set of prediction results P;

[0006] The pre-trained target detection model used was GroundingDINO.

[0007] Step 2: For each prediction pi∈P, perform manual annotation to obtain the feedback dataset H, where H contains the category labels of the detection boxes, where 1 represents a good target and -1 represents a bad target;

[0008] Step 3: Train reinforcement learning model R using the feedback dataset H;

[0009] In this process, a reward function r(pi) is defined, where r(pi) = 1 indicates that pi is a good objective, and r(pi) = -1 indicates that pi is a bad objective. The policy gradient method is used to update the parameter θ of R, and the update formula is:

[0010]

[0011] Where α is the learning rate, and J(θ) is the method for maximizing the expected cumulative reward, specifically... ∇θJ(θ) is the gradient of the expected cumulative reward J(θ) with respect to the parameter θ.

[0012] Step 4: Use D to predict the i-th image of the unlabeled field data, and use R to extract the target from the prediction result. Then fine-tune D by using the gradient descent algorithm and updating the parameters of D with the loss function L to minimize the loss.

[0013] The loss function L for the fine-tuning training is:

[0014]

[0015] Where λ1, λ2, and λ3 are weight parameters;

[0016]

[0017] Where z i For text features, z i + For the features of the corresponding image patch, sim is the cosine similarity function for calculation, and τ is the learnable temperature parameter;

[0018]

[0019] Where f is the image coding function, x i aug For the data augmented image x i .

[0020]

[0021] Where g is the classification header, y i Let be the label of image i, and CE be the cross-entropy loss function.

[0022] Step 5: Repeat step 4 to finally train the detection model D' for the target scene.

[0023] Compared with existing technologies, the beneficial effects of this invention are as follows: by using a general pre-trained model and then using reinforcement learning based on human feedback to fine-tune the model, it meets the needs of different scenarios, avoids retraining the detection model from scratch, reduces training time, reduces the number of manually labeled images, and helps to quickly adapt to new scenarios. Detailed Implementation

[0024] To clarify the technical solution and working principle of the present invention, the embodiments of this disclosure will be described in further detail below. All the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be elaborated upon here.

[0025] The terms "step 1," "step 2," "step 3," and similar descriptions used in the specification and claims of this application are for distinguishing similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those described herein.

[0026] This disclosure provides a target detection method incorporating reinforcement learning with human feedback, the method comprising the following steps:

[0027] Step 1: Collect field data using the pre-trained target detection model GroundingDINO, denoted as D, and use the trained D to predict the data to obtain the prediction result set P.

[0028] Besides GroundingDINO, other closed-set target detectors can also be used as alternatives.

[0029] Step 2: For each prediction pi∈P, manually label it as a good target of interest or a bad target of no interest, and obtain the feedback dataset H. H contains the category labels of the detection boxes, where 1 represents a good target and -1 represents a bad target.

[0030] Step 3: Train a reinforcement learning model R using the feedback dataset H. During this process, define a reward function r(pi), where r(pi) = 1 indicates that pi is a good target, and r(pi) = -1 indicates that pi is a bad target. Our goal is to maximize the expected cumulative reward: The parameter θ of R is updated using the policy gradient method, and the update formula is as follows:

[0031]

[0032] Where α is the learning rate, and ∇θJ(θ) is the gradient of the expected cumulative reward J(θ) with respect to the parameter θ.

[0033] Step 4: Use D to predict the i-th image of the unlabeled field data, and use R to extract the target from the prediction results. Then, fine-tune D, update the parameters of D to minimize the loss. The calculation steps for the training loss L are as follows:

[0034]

[0035] Where z iFor text features, z i + For the features of the corresponding image patch, sim is the cosine similarity function for calculation, and τ is the learnable temperature parameter.

[0036]

[0037] Where f is the image coding function, x i aug For the data augmented image x i .

[0038]

[0039] Where g is the classification header, y i Let be the label of image i, and CE be the cross-entropy loss function.

[0040]

[0041] Where λ1, λ2, and λ3 are weight parameters.

[0042] Use the gradient descent algorithm to update the parameters of D with L.

[0043] Step 5: Repeat step 4 to finally train the detection model D' for the target scene.

[0044] The present invention has been described above by way of example. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other situations without improvement or equivalent substitution, are all within the protection scope of the present invention.

Claims

1. A method for object detection combining reinforcement learning with human feedback, the method comprising: The method comprises the following steps: Step 1, collect n pieces of field image data, denoted as x i , i ∈ [1, 2, 3, …, n], use a general pre-trained target detection model D to make a prediction, and obtain a prediction result set P; Step 2: For each prediction pi in P, artificial labeling is performed to obtain a feedback data set H, H containing the class label of the detection frame, wherein 1 represents a good target and -1 represents a bad target; Step 3: training a reinforcement learning model R using the feedback data set H; Step 4: using D to predict the i-th unlabeled field data image, and using R to extract the good target in the prediction result, then fine-tuning D, using a gradient descent algorithm, and using a loss function L to update the parameters of D to minimize the loss; Step 5: repeat step 4 to finally train a detection model D' for the target scene; The loss function L of the fine-tuning training is: , Wherein, λ1, λ2, λ3 are weight parameters; , where z i is the text feature, z i + is the feature of the corresponding image patch, sim is the cosine similarity function, and τ is a learnable temperature parameter; , where f is an image encoding function, x i aug is the data-augmented image x i, , where g is the classification head, y i is the label of image i, and CE is the cross-entropy loss function. 2.The method of claim 1, wherein, The pre-trained target detection model selects GroundingDINO. 3.The method of claim 2, wherein, In the process of training the reinforcement learning model R in step 3, a reward function r(pi) is defined, wherein r(pi)=1 represents that pi is a good target, r(pi)=-1 represents that pi is a bad target, the parameters θ of R are updated using a policy gradient method, and the update formula is: , where a is the learning rate, J(0) is the expected cumulative reward to be maximized, and ∇0J(0) is the gradient of the expected cumulative reward J(0) with respect to the parameters 0.

Images