Dense target detection method and device based on large model and expert guidance

Abstract

The invention discloses a dense target detection method and device based on a large model and expert guidance. According to the method, preliminary target detection is carried out on image data by means of the detection capability of an image large model; and performing manual auditing and supplementing on the preliminary labeling result, and taking complete label data as a training target or a reward signal of the reinforcement learning model. Then, the training process of reinforcement learning is guided based on experts, a model training strategy is optimized, the convergence speed of the model is increased, and the model obtains the dense small target image detection capability; the image large model can be a target detection model or an image segmentation model, and a detection result or a segmentation result is converted into a reinforcement learning label. The reinforcement learning decision model can be suitable for general reinforcement learning algorithms such as DQN and PPO. The trained reinforcement learning model can realize dense target labeling of new image data, and the detection capability of the model in dense small targets is improved.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and in particular to a method and device for dense target detection based on a large model and expert guidance. Background Art

[0002] The rapid development of large-scale model technology has provided powerful intelligent support for solving complex problems, demonstrating exceptional performance in image and video segmentation. Representative examples include the Segment Anything Model (SAM), released in 2023, and the UniVS model, proposed in 2024. The former generates segmentation masks using prompts, while the latter utilizes a unified video mask decoder to process image and prompt information, improving segmentation accuracy.

[0003] However, in the task of automatic labeling of dense small objects, current large models perform insufficiently for the following reasons:

[0004] 1. Resolution reduction: When convolution and pooling layers process small objects, the resolution decreases, resulting in reduced recognition performance.

[0005] 2. Insufficient data: There is less data related to small objects, and the model's global attention mechanism tends to ignore subtle features.

[0006] 3. Anchor box mismatch: The size of the anchor box does not match the actual size of the small object, resulting in missed detection and false detection problems.

[0007] Reinforcement learning can gradually detect localized image information, reducing the loss of image information during convolutional downsampling. This can address the difficulty large models face in fully recognizing densely packed small objects or other complex scenarios. Reinforcement learning training often presents difficulties, such as difficulty learning useful information. Furthermore, balancing short-term and long-term goals during training is difficult. Therefore, combining the processing power of large models with the advantages of reinforcement learning is feasible.

[0008] Expert-guided reinforcement learning accelerated training is a method that combines expert knowledge and experience with reinforcement learning algorithms. It aims to improve the convergence rate of intelligent agents during training and enhance their detection capabilities in complex environments during inspection. This method, through expert guidance, helps intelligent agents learn more effectively in complex environments and promotes knowledge transfer and generalization across multiple categories of inspection tasks. In practical applications, it mainly involves bubble counting in chemical reactions and product inspection in factory workshops. In bubble counting technology for chemical reactions, traditional techniques have difficulty accurately counting the radius and number of bubbles in overlapping areas. In product quality inspection in factory workshops, the detection statistics of existing target detection models are inaccurate, and reinforcement learning is needed to perform fine detection of image data to achieve complete target detection. Summary of the Invention

[0009] The purpose of the present invention is to address the problems of difficulty in dense target detection and slow convergence of reinforcement learning training. The present invention proposes a dense target detection algorithm based on a large model and expert guidance.

[0010] The object of the present invention is achieved through the following technical solution: a dense target detection method based on a large model and expert guidance, the method comprising:

[0011] S1. Use the large model to pre-process the reference image to obtain the position and shape information of the small target, and convert the output of the large model into label data;

[0012] S2. Manually re-label the data missed by the large model detection; use the labeled data set as a reward for reinforcement learning;

[0013] S3. Collect data from the reference image and train the agent using the reinforcement learning method. During the training process, the reinforcement learning environment is updated each time to perceive the image according to the agent's detection range, and the collected data is stored in the experience pool.

[0014] S4: For agents that have not generated new labeled data after the number of operations reaches a set threshold, an expert guidance strategy is added to learn strategies that favor greater rewards, ultimately obtaining a well-trained model.

[0015] S5. Use the trained model to label the images that need to be labeled to obtain label data.

[0016] Furthermore, the use of the large model to pre-process the reference image to obtain the position and shape information of the small target specifically includes: pre-training the SAM / YOLO large model, freezing the learned task module part during the pre-training process, and using the pre-trained model to pre-label the data.

[0017] Furthermore, the use of the labeled data set as a reward for reinforcement learning specifically includes ordering the reward position, and the intelligent agent locks its perspective within a set angle range during each training, identifying and locking the target closest to its current position.

[0018] Furthermore, the data collection for the reference image is specifically as follows: the reference image is cut into sub-images according to the viewing range of the agent and the center position of the agent, and the current sub-image is used as the input state of the agent decision model; the input states are divided into three types: the agent is moving towards the target point, the agent moves to the target position to adjust the shape, and the agent completes the shape adjustment and is successfully marked.

[0019] Furthermore, in the process of training the agent using the reinforcement learning method, the Bellman equation of the action-value function is used to update the target network each time, so that the label data has reward information, which is calculated by the following formula:

[0020]

[0021] in: Is in state Take action Instant rewards, Is to perform an action Then transfer to the next state, is the discount factor, is the target network, which is the same as the current Q network, but with fixed parameters, and is updated from the current Q network every certain number of steps. is the action value function of the target network, used to calculate the next state Take action The maximum expected return when Is in state Next, select The biggest move .

[0022] Furthermore, in the process of training the agent using the reinforcement learning method, the loss function is designed as follows:

[0023] The agent decision network uses the action value function of the current network Close to target value , the following loss is used to measure the difference between the two:

[0024]

[0025] in, is the current prediction value of the Q network, is the target value calculated by the target network, are the parameters of the Q network.

[0026] Furthermore, the strategy of adding expert guidance to the agent that has not generated new labeled data after the number of operations reaches a set threshold includes:

[0027] Counting the number of actions taken by an agent in a reinforcement learning task , and set an upper limit . Each time a target is marked, the operation count starts. If achieve When no new labeled data is generated, the agent's actions are guided by experts;

[0028] Calculate the optimal strategy between the current state and the reward position through the contour distance, and use the strategy as the simulation strategy data;

[0029]

[0030] in: is the strategy calculation module, The value function is calculated by the reward position and the agent position. The closer the distance, the greater the reward value, and vice versa. is the intersection-over-combination ratio between the agent and the reward target, is a hyperparameter, preventing the value function The value is too large;

[0031] The agent simulates operations according to the action sequence, obtains rewards for long-term goals, and puts them into the experience pool;

[0032] By prioritizing experience replay, the sampling probability of expert-guided samples is increased, so that the model can prioritize learning long target samples and data samples after expert guidance.

[0033] On the other hand, this specification also provides a dense target detection device based on a large model and expert guidance, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the dense target detection method based on a large model and expert guidance.

[0034] On the other hand, this specification also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the dense target detection method based on a large model and expert guidance.

[0035] Beneficial effects of the present invention:

[0036] The present invention proposes a dense target detection algorithm based on a large model and expert guidance. The current detection capability of dense small targets lacks data samples of dense small target scenes. Usually, a large image model is used to detect dense small targets. The large model has limited feature extraction capabilities for small pixels because the large model uses multi-layer downsampling in the image extraction process, which causes problems such as feature loss and low feature attention in the feature extraction process. The present invention first uses the detection capability of the large image model to pre-process the image, and then uses a manual annotation method to complete the target detection results as learning samples for reinforcement learning; in response to the problem that reinforcement learning is difficult to train, the present invention designs a training method with an expert-guided strategy to optimize the quality of the reinforcement learning experience pool, so that the reinforcement learning model can quickly learn to obtain long target rewards and long target sample data, thereby accelerating the convergence speed of the reinforcement learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A learning framework for expert guidance;

[0038] Figure 2 To mark the target locking module;

[0039] Figure 3 This is a schematic diagram of the reinforcement learning framework;

[0040] Figure 4 This is a graph showing the change of action logic values ​​over iterations during training without expert guidance;

[0041] Figure 5 The number of annotations simulated for training without expert guidance varies with the number of iterations;

[0042] Figure 6 This is a graph showing the change of action logic values ​​with iterations during expert-guided training;

[0043] Figure 7 The number of annotations in the expert-guided training simulation changes with the number of iterations;

[0044] Figure 8 Schematic diagram of a dense target detection device based on a large model and expert guidance provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0046] like Figure 1 and Figure 3As shown, the present invention provides an automatic labeling algorithm based on a large model and expert guidance. The main idea is to complete preprocessing based on a large model related to a specific task. If it is an image target detection task, the corresponding target detection model (such as YOLO and SAM related models) completes the preprocessing. If it is a graphic and text cross-modal related task, the CLIP related model is used to complete the preprocessing. The preprocessing result data and the manually labeled result data are both used as learning data for reinforcement learning. The use of expert guidance in training reinforcement learning speeds up training efficiency. The present invention can capture tiny features in image information and a faster and more effective training reinforcement learning method. The overall technical solution of the present invention (taking dense small target labeling as an example) includes the following steps:

[0047] S1, based on the preprocessing of the large model, uses models such as SAM / YOLO to preprocess the reference image to obtain the position and shape information of small targets, formulate rewards, and convert the output results of the large model into label data.

[0048] Models like SAM / YOLO have limited detection capabilities for densely packed small objects, often missing or incorrectly detecting targets. Current mainstream large models typically use a series of convolutional and pooling layers to extract high-level features, which reduces the resolution of small objects and blurs them. The size of the anchor boxes also mismatches the actual size of small objects, leading to misdetection or missed detections. The pre-trained large model needs to be fine-tuned. A small dataset is prepared and then used to train the pre-trained model and the small dataset together. During fine-tuning, some modules of the pre-trained model are frozen to prevent degradation of the learned tasks. The fine-tuned model is then used to pre-label the data to improve the quality of the annotations.

[0049] Specifically, for dense small object detection, it is necessary to select a large image processing model such as an object detection model or an image segmentation model. For pre-labeling, large image models such as SAM, U-Net, YOLO, and UniVS are used. For example, the SAM model is used to segment the image to be labeled.

[0050] The mathematical method of contour distance is used to calculate the segmented data features after SAM segmentation. The shape and key point position of each mask are converted to the same form as the annotation data through the segmentation mask data output by SAM as the annotation result;

[0051] S2, manually completes the pre-processed data of the large model. The large model has the problem of missed detection. The remaining targets in the reference image are manually labeled to achieve 100% labeling effect, and the labeled data is made into a data set for training the reinforcement learning model.

[0052] S3, the training data set is used as a reward for reinforcement learning. This invention takes dense small target labeling as an example, and the labeled data in the data set is used as the reward position. It is necessary to order the reward position to improve the stability of learning. The agent needs to lock into an angle range each time, refresh the locking range according to the position of the agent each time, and identify and lock the target closest to its current position (such as Figure 2 As shown): Calculate the nearest target within the angle and range based on the current agent position, and use this target as the next reward target. At this time, the reward value of this reward target is the largest (after each target is marked, the next target is calculated and locked). The locking bubble mechanism is set, as shown Figure 2 shown. Figure 2 Set the hyperparameter viewing range in , The upper and lower angles of the line directly to the right of the agent (the angle size is a hyperparameter. When the target density is high, the angle should be appropriately reduced to reduce the difficulty of agent training).

[0053] S4, build the environment according to the actual application scenario. When building the environment for automatic labeling of dense small objects, in order to avoid overfitting and poor generalization problems, it is necessary to sample the data (collect effective information based on the position and state of the agent) and crop the sub-image of the reference image according to the agent's viewing range and the center position of the agent.

[0054] S41 collects data from the reference image. The reinforcement learning environment divides the input image into several sub-images based on the agent's perception range. The current sub-image is used as the input state of the agent's decision model. There are three states: the agent is moving towards the target point, the agent has moved to the target location and adjusted its shape, and the agent has completed the shape adjustment and successfully labeled it.

[0055] S42, each update uses the Bellman equation of the action-value function to make the label data have reward information and update the target network. It can be calculated by the following formula:

[0056]

[0057] in:

[0058] - Is in state Take action Instant rewards.

[0059] - Is to perform an action Then transfer to the next state.

[0060] - is the discount factor.

[0061] - is the target network, which is the same as the current Q network, but with fixed parameters, and is updated from the current Q network every certain number of steps.

[0062] - is the action value function of the target network, used to calculate the next state Take action The maximum expected return when .

[0063] - Is in state Next, select The biggest move .

[0064] S43, design the loss function and update the network parameters. The loss is designed as follows:

[0065] The agent decision network uses the action value function of the current network Close to target value , the following loss is used to measure the difference between the two:

[0066]

[0067] in:

[0068] - is the current prediction value of the Q network.

[0069] - is the target value calculated by the target network.

[0070] - are the parameters of the Q network.

[0071] S5, during the training of the agent annotation process, for the parts with slow learning speed, an expert-guided training strategy is added to interfere with the action strategy, optimize the action strategy, and improve the learning efficiency and the quality of the experience pool;

[0072] S51, the agent is placed in a marked range with slower learning efficiency, and the expert obtains the optimal strategy through calculation;

[0073] S511, Counting the number of agent actions in reinforcement learning tasks , and set an upper limit . Each time a target is marked, the operation count starts. If achieve When no new labeled data is generated, the agent's actions are guided by experts;

[0074] S512, calculating the optimal strategy between the current state and the reward position through the contour distance, and using the strategy as the simulation strategy data;

[0075]

[0076] - Strategy calculation module

[0077] - The value function is calculated by the reward position and the agent position. The closer the distance, the greater the reward value, and vice versa.

[0078] - is the intersection-over-combination ratio between the agent and the reward target

[0079] - is a hyperparameter, preventing the value function The value is too large

[0080] S52, the agent simulates the operation according to the action sequence, obtains the reward of the long goal, and puts it into the experience pool;

[0081] S53, by prioritizing experience playback, the sampling probability of expert-guided samples is increased, so that the model can prioritize learning long target samples and expert-guided data samples, such as Figure 1 As shown in the figure, the training ends when reinforcement learning completes labeling of all objects in the image.

[0082] In the reinforcement learning process, the learning efficiency of the agent is closely related to its exploration ability, environmental feedback, and knowledge acquisition method. By comparing the agent's behavioral decision-making pattern and data labeling efficiency under the two strategies of non-expert guided training and expert guided training, we can verify the positive effect of expert guidance on the reinforcement learning process. Figure 4 As shown in the figure, in the absence of expert-guided training, the agent's action logic value changes relatively slowly, indicating that the model is unable to effectively acquire high-value strategies in the early stages and the knowledge learned is relatively limited. This phenomenon reflects the exploration and utilization problem in reinforcement learning, that is, in the absence of high-quality guidance, it is difficult for the agent to quickly master the optimal strategy. Figure 6 As shown in the figure, after adding expert-guided training, the action logic value of the agent changes rapidly in the early stage, and can quickly adapt to the current simulation environment and learn more valuable knowledge. This shows that the expert-guided mechanism can significantly improve the decision-making efficiency of the agent and accelerate the strategy optimization process. In addition, in reinforcement learning, the quality and quantity of labeled data have a key impact on the learning effect of the agent. Figure 5 As shown in , without expert-guided training, the growth of labeled data is relatively slow, indicating that it is difficult for the agent to efficiently obtain effective data during autonomous exploration, resulting in a relatively lengthy learning process. Figure 7As shown, under expert-guided training, the amount of labeled data increases by over five times compared to non-expert-guided training within the same number of iterations, demonstrating more efficient data acquisition and utilization. This result demonstrates that expert-guided training can optimize the agent's exploration strategy, enabling it to more quickly identify key states, improve data utilization, and accelerate convergence. Comprehensive experimental results show that expert-guided reinforcement learning can effectively improve the agent's decision-making efficiency and data utilization. By introducing expert guidance, the agent can adapt to the environment more quickly, optimize its decision-making strategy, and improve learning efficiency. Data labeling efficiency is significantly improved, allowing the agent to acquire key experience more quickly, thereby reducing ineffective exploration. Furthermore, the reinforcement learning model can form a better strategy early in the training phase, shortening training time and improving the model's generalization ability. Therefore, in reinforcement learning applications, the appropriate introduction of expert guidance mechanisms, combined with human knowledge and autonomous learning capabilities, can significantly improve the training effectiveness of the agent and provide more optimal solutions for efficient decision-making in complex environments.

[0083] S6: Use the trained model to label the images that need to be labeled to obtain label data.

[0084] Corresponding to the aforementioned embodiment of a dense target detection method based on a large model and expert guidance, the present invention also provides an embodiment of a dense target detection device based on a large model and expert guidance.

[0085] See also Figure 8 An embodiment of the present invention provides a dense target detection device based on a large model and expert guidance, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a dense target detection method based on a large model and expert guidance in the above embodiment.

[0086] The embodiment of a dense target detection device based on a large model and expert guidance provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 8 As shown in the figure, a hardware structure diagram of a dense target detection device based on a large model and expert guidance provided by the present invention is located in any device with data processing capabilities, except Figure 8In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0087] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0088] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0089] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a dense target detection method based on a large model and expert guidance in the above embodiment is implemented.

[0090] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0091] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the dense target detection method based on a large model and expert guidance.

[0092] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims.

[0093] It should be understood that the above general description and the detailed description that follows are exemplary and explanatory only and do not limit the present application. The present application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope of the present application. The scope of the present application is limited only by the appended claims.

Claims

1. A dense target detection method based on a large model and expert guidance, characterized in that: The method includes: S1. Use the large model to pre-process the reference image to obtain the position and shape information of the small target, and convert the output of the large model into label data; S2. Manually re-label the data missed by the large model detection; use the labeled data set as a reward for reinforcement learning; S3. Collect data from the reference image and train the agent using the reinforcement learning method. During the training process, the reinforcement learning environment is updated each time to perceive the image according to the agent's detection range, and the collected data is stored in the experience pool. S4: For agents that have not generated new labeled data after the number of operations reaches a set threshold, an expert guidance strategy is added to learn strategies that favor greater rewards, ultimately obtaining a well-trained model. S5. Use the trained model to label the images that need to be labeled to obtain label data.

2. The method for dense target detection based on large models and expert guidance according to claim 1, characterized in that: The use of a large model to pre-process the reference image to obtain the position and shape information of the small target specifically includes: pre-training the SAM / YOLO large model, freezing the learned task module part during the pre-training process, and using the pre-trained model to pre-label the data.

3. The method for dense target detection based on large models and expert guidance according to claim 1, characterized in that: The use of the labeled data set as a reward for reinforcement learning specifically includes ordering the reward position. During each training session, the intelligent agent locks its perspective within a set angle range and identifies and locks onto the target closest to its current position.

4. The method for dense target detection based on large models and expert guidance according to claim 1, characterized in that: The data collection for the reference image is specifically as follows: the reference image is cut into sub-images according to the viewing range of the agent and the center position of the agent, and the current sub-image is used as the input state of the agent decision model; the input states are divided into three types: the agent is moving towards the target point, the agent moves to the target position to adjust the shape, and the agent completes the shape adjustment and is successfully marked.

5. The method for dense target detection based on large models and expert guidance according to claim 1, characterized in that: In the process of training the agent using the reinforcement learning method, the Bellman equation of the action-value function is used to update the target network each time, so that the label data has reward information, which is calculated by the following formula: ； in: Is in state Take action Instant rewards, Is to perform an action Then transfer to the next state, is the discount factor, is the target network, which is the same as the current Q network, but with fixed parameters, and is updated from the current Q network every certain number of steps. is the action value function of the target network, used to calculate the next state Take action The maximum expected return when Is in state Next, select The biggest move .

6. The method for dense target detection based on large models and expert guidance according to claim 5, characterized in that: In the process of training the agent using the reinforcement learning method, the loss function is designed as follows: The agent decision network uses the action value function of the current network Close to target value , the following loss is used to measure the difference between the two: in, is the current prediction value of the Q network, is the target value calculated by the target network, are the parameters of the Q network.

7. The method for dense target detection based on large models and expert guidance according to claim 1, characterized in that: The strategy of adding expert guidance to the agent that has not generated new labeled data after the number of operations reaches a set threshold includes: Counting the number of actions taken by an agent in a reinforcement learning task , and set an upper limit ; Each time a target is marked, the operation count starts. If achieve When no new labeled data is generated, the agent's actions are guided by experts; Calculate the optimal strategy between the current state and the reward position through the contour distance, and use the strategy as the simulation strategy data; ； in: is the strategy calculation module, The value function is calculated by the reward position and the agent position. The closer the distance, the greater the reward value, and vice versa. is the intersection-over-union ratio between the agent and the reward target, is a hyperparameter, preventing the value function The value is too large; The agent simulates operations according to the action sequence, obtains rewards for long-term goals, and puts them into the experience pool; By prioritizing experience replay, the sampling probability of expert-guided samples is increased, so that the model can prioritize learning long target samples and expert-guided data samples.

8. A dense target detection device based on a large model and expert guidance, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a dense target detection method based on a large model and expert guidance according to any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a dense target detection method based on a large model and expert guidance according to any one of claims 1 to 7 is implemented.

Images