A method and apparatus for dense target detection based on large models and expert guidance

Abstract

This invention discloses a method and apparatus for dense target detection based on a large image model and expert guidance. The method utilizes the detection capabilities of a large image model to perform preliminary target detection on image data; the preliminary annotation results are manually reviewed and supplemented, and the complete label data is used as the training target or reward signal for a reinforcement learning model. Subsequently, expert-guided reinforcement learning training is employed to optimize the model training strategy, accelerate model convergence, and enable the model to acquire the ability to detect dense small target images. The large image model can be a target detection model or an image segmentation model, converting the detection or segmentation results into reinforcement learning labels. The reinforcement learning decision model is applicable to general reinforcement learning algorithms such as DQN and PPO. The trained reinforcement learning model can achieve dense target annotation on new image data, improving the model's detection capability among dense small targets.

Description

Technical Field

[0001] This invention relates to the field of target detection, and in particular to a dense target detection method and apparatus based on a large model and expert guidance. Background Technology

[0002] The rapid development of large model technology has provided powerful intelligent support for solving complex problems, especially demonstrating outstanding performance in image and video segmentation. The Segment Anything Model (SAM) released in 2023 and the UniVS model proposed in 2024 are representative examples. The former generates segmentation masks through prompts, while the latter uses a unified video mask decoder to process image and prompt information, thus improving the accuracy of segmentation tasks.

[0003] However, current large models perform poorly in tasks involving the automatic annotation of dense, small targets, mainly due to the following reasons:

[0004] 1. Reduced resolution: When convolutional and pooling layers process small targets, the reduced resolution leads to decreased recognition performance.

[0005] 2. Insufficient data: There is a lack of data related to small targets, and the model's global attention mechanism is prone to ignoring subtle features.

[0006] 3. Anchor frame mismatch: The anchor frame size does not match the actual size of the small target, leading to missed detections and false detections.

[0007] Reinforcement learning enables localized, stepwise detection of image information, reducing information loss during convolutional downsampling. It can address the challenge of large-scale object detection models struggling to comprehensively identify densely labeled small targets or other complex scenarios. However, training reinforcement learning often faces difficulties, such as the model failing to learn useful information. Furthermore, it struggles to balance short-term and long-term objectives during training. Therefore, combining the processing power of large-scale models with the advantages of reinforcement learning presents a viable option.

[0008] Expert-guided reinforcement learning (ABL) accelerated training is a method that combines expert knowledge and experience with reinforcement learning algorithms. It aims to improve the convergence speed of the agent during training and enhance its detection capabilities in complex environments. This method, guided by experts, helps the agent learn more effectively in complex environments and promotes knowledge transfer and generalization across multiple detection categories. In practical applications, it primarily involves bubble counting in chemical reactions and product inspection in factory workshops. In bubble counting techniques for chemical reactions, traditional methods struggle to accurately count the radius and number of bubbles in overlapping areas. In product quality inspection in factory workshops, existing target detection models are inaccurate in their statistical counts, necessitating the use of reinforcement learning for refined detection of image data to achieve complete target detection. Summary of the Invention

[0009] The purpose of this invention is to address the difficulties in dense target detection and the slow convergence speed of reinforcement learning training. This invention proposes a dense target detection algorithm based on a large model and expert guidance.

[0010] The objective of this invention is achieved through the following technical solution: a dense target detection method based on a large model and expert guidance, the method comprising:

[0011] S1. Use a large model to preprocess the reference image to obtain the position and shape information of the small target, and convert the output of the large model into label data;

[0012] S2. Manually re-annotate the data that the large model missed; use the annotated dataset as a reward for reinforcement learning.

[0013] S3. Collect data from the reference image and train the agent using reinforcement learning. During the training process, the reinforcement learning environment is updated each time to perceive the image according to the agent's detection range, and the collected data is stored in the experience pool.

[0014] S4. For agents that have not generated new labeled data after the number of operations has reached a set threshold, add an expert guidance strategy to learn a strategy that is biased towards greater rewards, and finally obtain a well-trained model.

[0015] S5. Use the trained model to annotate the images that need to be labeled to obtain label data.

[0016] Furthermore, the step of using a large model to preprocess the reference image to obtain the position and shape information of the small target specifically includes: pre-training the SAM / YOLO large model, freezing the learned task modules during the pre-training process, and using the pre-trained model to pre-label the data.

[0017] Furthermore, the use of the labeled dataset as a reward for reinforcement learning specifically includes ordering the reward positions, locking the agent's viewpoint within a set angle range during each training session, and identifying and locking onto the target closest to its current position.

[0018] Furthermore, the specific steps of collecting data from the reference image are as follows: cropping the reference image into sub-images according to the agent's viewpoint and the agent's center position, and using the current sub-image as the input state of the agent's decision-making model; the input state is divided into three types: the agent is moving towards the target point, the agent moves to the target position and adjusts its shape, and the agent completes shape adjustment and successfully labels the image.

[0019] Furthermore, during the training of the agent using reinforcement learning, each update of the target network employs the Bellman equation of the action value function, ensuring that the labeled data contains reward information, calculated using the following formula:

[0020]

[0021] in: It is in state Take action below Instant rewards It is to perform an action The next state after that, It is a discount factor. This is the target network, which is the same as the current Q-network, but its parameters are fixed and are updated from the current Q-network every certain number of steps. It is the action-value function of the target network, used to calculate the next state. China takes action Maximum expected return at that time It is in state Below, the choice makes Biggest movement .

[0022] Furthermore, in the process of training the agent using reinforcement learning methods, the loss function is designed as follows:

[0023] The agent decision network uses the action value function of the current network. Approaching the target value The following loss is used to measure the difference between the two:

[0024]

[0025] in, This is the current prediction value of the Q network. The target value is calculated through the target network. These are the parameters of the Q-network.

[0026] Furthermore, the strategy of adding expert guidance to agents that fail to generate new labeled data after reaching a set threshold number of operations includes:

[0027] In reinforcement learning tasks, the number of actions performed by the agent is counted. And set an upper limit. After each target is labeled, start counting the operations. achieve If no new labeled data is generated within the upper limit, expert guidance is provided for the agent's actions.

[0028] The optimal policy between the current state and the reward position is calculated by using the contour distance, and the policy is used as the policy data for simulation.

[0029]

[0030] in: For the strategy calculation module, The reward function is calculated using the reward location and the agent's location; the closer the distance, the larger the reward value, and vice versa. The intersection-union ratio between the agent and the reward objective. Hyperparameters prevent value functions The value is too large;

[0031] The agent simulates operations based on action sequences, obtains rewards for long-term goals, and adds them to the experience pool;

[0032] By prioritizing experience replay, the sampling probability of expert-guided samples is increased, enabling the model to learn long target samples and expert-guided data samples first.

[0033] On the other hand, this specification also provides a dense target detection device based on a large model and expert guidance, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the aforementioned dense target detection method based on a large model and expert guidance.

[0034] On the other hand, this specification also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned dense target detection method based on a large model and expert guidance.

[0035] The beneficial effects of this invention are:

[0036] This invention proposes a dense target detection algorithm based on a large model and expert guidance. Current detection capabilities for dense small targets lack sufficient data samples for such scenarios. Typically, large image models are used for dense small target detection; however, these models have limited feature extraction capabilities for small pixels because they employ multi-layer downsampling during image extraction, leading to feature loss and low feature attention. This invention first utilizes the detection capabilities of a large image model for image preprocessing, then uses manual annotation to complete the target detection results, serving as learning samples for reinforcement learning. To address the difficulty in training reinforcement learning, this invention designs an expert-guided training strategy to optimize the quality of the reinforcement learning experience pool, enabling the reinforcement learning model to quickly learn long-term target rewards and long-term target sample data, thereby accelerating the convergence speed of the reinforcement learning model. Attached Figure Description

[0037] Figure 1 A learning framework guided by experts;

[0038] Figure 2 For marking target locking module;

[0039] Figure 3 A diagram illustrating the principles of a reinforcement learning framework;

[0040] Figure 4 A graph showing the change of action logic values ​​during iterative training without expert guidance.

[0041] Figure 5 The graph shows the change in the number of labeled elements in an unguided training simulation as a function of the number of iterations.

[0042] Figure 6 A graph showing the change of action logic values ​​during expert-guided training as iterates;

[0043] Figure 7 The graph shows the change in the number of labeled elements in an expert-guided training simulation as a function of the number of iterations.

[0044] Figure 8 This is a schematic diagram of a dense target detection device based on a large model and expert guidance provided in an embodiment of the present invention. Detailed Implementation

[0045] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0046] like Figure 1 and Figure 3As shown, this invention provides an automatic annotation algorithm based on large models and expert guidance. The main idea is to perform preprocessing using a large model relevant to the specific task. For image object detection tasks, the corresponding object detection model (such as YOLO and SAM-related models) is used for preprocessing; for cross-modal image-text tasks, the CLIP-related model is used. Both the preprocessed data and the manually annotated data are used as training data for reinforcement learning. Expert guidance is used to accelerate training efficiency. This invention can capture subtle features in image information and provides a faster and more effective reinforcement learning training method. The overall technical solution of this invention (taking dense small object annotation as an example) includes the following steps:

[0047] S1, based on the preprocessing of the large model, uses models such as SAM / YOLO to preprocess the reference image to obtain the position and shape information of small targets, and formulates the reward. The output of the large model is converted into label data.

[0048] Models like SAM / YOLO have limited ability to detect densely packed small objects, often resulting in missed or incorrect detections. Current mainstream large-scale models typically use a series of convolutional and pooling layers to extract high-level features, which reduces the resolution of small objects, making them blurry. Furthermore, the anchor box size may not match the actual size of the small objects, leading to detection errors or missed detections. Therefore, fine-tuning of the pre-trained large-scale model is necessary. A small dataset should be used to train the pre-trained model alongside this small dataset. During fine-tuning, freezing certain modules of the pre-trained model can prevent degradation of learned tasks. Finally, the fine-tuned model should be pre-labeled to enhance the labeling effect.

[0049] Specifically, for dense small object detection requirements, it is necessary to select a large image processing model, such as an object detection model or an image segmentation model, and use large image models such as SAM, U-Net, YOLO, and UniVS for pre-annotation processing. Taking the SAM model as an example, the SAM model is used to segment the image that needs to be annotated.

[0050] The segmented data features are calculated using the contour distance mathematical method on the SAM segmented data. The shape and keypoint positions of each mask are converted into the same format as the labeled data using the SAM output segmentation mask data as the annotation result.

[0051] S2 involves manually completing the preprocessed data of the large model. The large model has a problem of missing detections, so the remaining targets in the reference image are manually labeled to achieve 100% labeling effect. The labeled data is then used as the dataset for training the reinforcement learning model.

[0052] S3 uses the training dataset as the reward for reinforcement learning. This invention takes dense small target annotation as an example, where the annotated data in the dataset is used as the reward position. The reward positions need to be ordered to improve the stability of the learning process. The agent needs to lock onto a specific angular range each time, refreshing the locking range based on the agent's position each time, and identifying and locking onto the target closest to its current position (e.g., [missing information]). Figure 2 (As shown): Calculate the nearest target within the angle and range based on the current agent's position, and use this target as the next reward target. At this point, the reward value for this target is maximized (after each target is labeled, the next target is calculated and locked). The lock bubble mechanism is set as follows: Figure 2 As shown. Figure 2 Setting the hyperparameter view range , The angle between the agent and the line directly to the right is the size of the vertical angle (the angle is a hyperparameter; when the target density is high, the angle should be reduced appropriately to reduce the training difficulty of the agent).

[0053] S4. When constructing an environment based on actual application scenarios and automatically labeling dense small targets, in order to avoid overfitting and poor generalization, it is necessary to sample the data (collect effective information based on the position and state of the agent) and crop the reference image into sub-images according to the agent's field of view and the agent's center position.

[0054] S41: Data is collected from the reference image. The reinforcement learning environment divides the input image into several sub-images based on the agent's perceived image range each time, using the current sub-image as the input state for the agent's decision-making model. There are three states: the agent is moving towards the target point, the agent moves to the target position and adjusts its shape, and the agent completes shape adjustment and successfully labels the image.

[0055] S42, Each update uses the Bellman equation of the action-value function, which imbues the label data with reward information. The target network is updated using the following formula:

[0056]

[0057] in:

[0058] - It is in state Take action below Instant rewards.

[0059] - It is to perform an action The state is then transitioned to.

[0060] - It is a discount factor.

[0061] - It is the target network, which is the same as the current Q network, but its parameters are fixed and are updated from the current Q network every certain number of steps.

[0062] - It is the action-value function of the target network, used to calculate the next state. China takes action The maximum expected return at that time.

[0063] - It is in state Below, the choice makes Biggest movement .

[0064] S43, Design the loss function to update the network parameters. The loss design is as follows:

[0065] The agent decision network uses the action value function of the current network. Approaching the target value The following loss is used to measure the difference between the two:

[0066]

[0067] in:

[0068] - This is the current prediction value of the Q network.

[0069] - It is the target value calculated through the target network.

[0070] - These are the parameters of the Q network.

[0071] S5, during the training of agent annotation, for the parts with slow learning speed, adds an expert-guided training strategy to interfere with the action strategy, optimize the action strategy, and improve learning efficiency and experience pool quality.

[0072] S51 allows experts to obtain the optimal strategy by calculating within a labeled range where the learning efficiency of the agent is relatively slow.

[0073] S511 counts the number of actions performed by the agent in a reinforcement learning task. And set an upper limit. After each target is labeled, start counting the operations. If achieve If no new labeled data is generated within the upper limit, expert guidance is provided for the agent's actions.

[0074] S512 calculates the optimal policy between the current state and the reward position using the contour distance, and uses the policy as the policy data for simulation;

[0075]

[0076] - For the strategy calculation module

[0077] - The reward function is calculated using the reward location and the agent's location; the closer the distance, the larger the reward value, and vice versa.

[0078] - The intersection-exchange ratio between the agent and the reward objective

[0079] - Hyperparameters prevent value functions The value is too large

[0080] S52, the agent simulates operations based on the action sequence, obtains the reward for the long goal, and puts it into the experience pool;

[0081] S53, by prioritizing experience replay, increases the sampling probability of expert-guided samples, enabling the model to prioritize learning long target samples and expert-guided data samples, such as... Figure 1 As shown. Training ends when reinforcement learning has labeled all objects within the image.

[0082] In reinforcement learning, the learning efficiency of an agent is closely related to its exploration ability, environmental feedback, and knowledge acquisition methods. By comparing the agent's behavioral decision-making patterns and data labeling efficiency under two strategies—unguided training and guided training—the positive role of expert guidance in the reinforcement learning process can be verified. Figure 4 As shown, without expert-guided training, the agent's action logic values ​​change relatively smoothly, indicating that the model struggles to effectively acquire high-value policies in the early stages, and its learned knowledge is limited. This phenomenon reflects the exploration and utilization challenge in reinforcement learning; that is, without high-quality guidance, the agent struggles to quickly master the optimal policy. And as... Figure 6 As shown, after expert-guided training, the agent's action logic values ​​change rapidly in the initial stage and can quickly adapt to the current simulation environment, learning more valuable knowledge. This indicates that the expert-guided mechanism can significantly improve the agent's decision-making efficiency and accelerate the policy optimization process. Furthermore, in reinforcement learning, the quality and quantity of labeled data have a crucial impact on the agent's learning performance. Figure 5 As shown, without expert-guided training, the labeled data grows relatively slowly, indicating that the agent struggles to efficiently acquire effective data during autonomous exploration, resulting in a lengthy learning process. However, as... Figure 7As shown, under expert-guided training, the amount of labeled data increases by more than 5 times compared to training without expert guidance within the same number of iterations, demonstrating more efficient data collection and utilization capabilities. This result verifies that expert-guided training can optimize the agent's exploration strategy, enabling it to identify key states more quickly, improve data utilization, and accelerate convergence. In summary, the experimental results demonstrate that expert-guided reinforcement learning can effectively improve the agent's decision-making efficiency and data utilization. By introducing expert guidance, the agent can adapt to the environment more quickly, optimize decision-making strategies, improve learning efficiency, and significantly enhance data labeling efficiency. This allows the agent to acquire key experiences more rapidly, thereby reducing ineffective exploration. Furthermore, the reinforcement learning model can form a superior strategy in the early training stages, shortening training time and improving the model's generalization ability. Therefore, in reinforcement learning applications, the reasonable introduction of expert guidance mechanisms, combined with human knowledge and autonomous learning capabilities, can significantly improve the training effect of the agent, providing better solutions for efficient decision-making in complex environments.

[0083] S6 uses the trained model to annotate the images that need to be labeled, thus obtaining the label data.

[0084] Corresponding to the aforementioned embodiment of a dense target detection method based on a large model and expert guidance, the present invention also provides an embodiment of a dense target detection device based on a large model and expert guidance.

[0085] See Figure 8 The present invention provides a dense target detection device based on a large model and expert guidance, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a dense target detection method based on a large model and expert guidance as described in the above embodiment.

[0086] The embodiment of the dense target detection device based on a large model and expert guidance provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 8 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is an example of a dense target detection device based on a large model and expert guidance provided by the present invention. (Except for...) Figure 8In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0087] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0088] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0089] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a dense target detection method based on a large model and expert guidance as described in the above embodiments.

[0090] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0091] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned dense target detection method based on a large model and expert guidance.

[0092] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0093] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A dense target detection method based on a large model and expert guidance, characterized in that, The method includes: S1. Use a large model to preprocess the reference image to obtain the position and shape information of the small target, and convert the output of the large model into label data; S2. Manually re-annotate the data that the large model missed; use the annotated dataset as a reward for reinforcement learning. S3. Collect data from the reference image and train the agent using reinforcement learning. During the training process, the reinforcement learning environment is updated each time to perceive the image according to the agent's detection range, and the collected data is stored in the experience pool. S4. For agents that have not generated new labeled data after the number of operations has reached a set threshold, add an expert guidance strategy to learn a strategy that is biased towards greater rewards, and finally obtain a well-trained model. S5. Use the trained model to annotate the images that need to be labeled to obtain label data.

2. The dense target detection method based on a large model and expert guidance according to claim 1, characterized in that, The method of using a large model to preprocess the reference image to obtain the position and shape information of the small target specifically includes: pre-training the SAM / YOLO large model, freezing the learned task modules during the pre-training process, and using the pre-trained model to pre-label the data.

3. The dense target detection method based on a large model and expert guidance according to claim 1, characterized in that, The use of the labeled dataset as a reward for reinforcement learning also includes ordering the reward positions, locking the viewpoint within a set angle range during each training session, and identifying and locking onto the target closest to its current position.

4. The dense target detection method based on a large model and expert guidance according to claim 1, characterized in that, The specific steps for collecting data from the reference image are as follows: cropping the reference image into sub-images according to the agent's viewpoint and the agent's center position, and using the current sub-image as the input state of the agent's decision-making model; the input state is divided into three types: the agent is moving towards the target point, the agent moves to the target position and adjusts its shape, and the agent completes shape adjustment and successfully labels the image.

5. The dense target detection method based on a large model and expert guidance according to claim 1, characterized in that, During the training of the agent using reinforcement learning, each update of the target network employs the Bellman equation of the action value function, ensuring that the labeled data contains reward information, calculated using the following formula: ； in: It is in state Take action below Instant rewards It is to perform an action The next state after that is transitioned to. It is a discount factor. This is the target network, which is the same as the current Q-network, but its parameters are fixed and are updated from the current Q-network every certain number of steps. It is the action-value function of the target network, used to calculate the next state. China takes action Maximum expected return at that time It is in state Below, the choice makes Biggest movement .

6. The dense target detection method based on a large model and expert guidance according to claim 5, characterized in that, In the process of training the agent using reinforcement learning methods, the loss function is designed as follows: The agent decision network uses the action value function of the current network. Approaching the target value The following loss is used to measure the difference between the two: in, This is the current prediction value of the Q network. The target value is calculated through the target network. These are the parameters of the Q network.

7. The dense target detection method based on a large model and expert guidance according to claim 1, characterized in that, The strategy of adding expert guidance to agents that fail to generate new labeled data after the number of operations reaches a set threshold includes: In reinforcement learning tasks, the number of actions performed by the agent is counted. And set an upper limit. After each target is labeled, start counting the operations. achieve If no new labeled data is generated within the upper limit, expert guidance is provided for the agent's actions. The optimal policy between the current state and the reward position is calculated by using the contour distance, and the policy is used as the policy data for simulation. ； in: For the strategy calculation module, The reward function is calculated using the reward location and the agent's location; the closer the distance, the larger the reward value, and vice versa. The intersection-union ratio between the agent and the reward objective. Hyperparameters prevent value functions The value is too large; The agent simulates operations based on action sequences, obtains rewards for long-term goals, and adds them to the experience pool; By prioritizing experience replay, the sampling probability of expert-guided samples is increased, enabling the model to learn long target samples and expert-guided data samples first.

8. A dense target detection device based on a large model and expert guidance, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a dense target detection method based on a large model and expert guidance as described in any one of claims 1-7.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a dense target detection method based on a large model and expert guidance as described in any one of claims 1-7.

Images