Active target positioning method, related equipment and storage medium

Abstract

The present invention provides an active target positioning method, related equipment, and storage medium, relating to the field of deep reinforcement learning technology. The method is applied to a VGG16 network and includes: step S1, inputting a to-be-processed image into a pre-trained VGG16 network; step S2, constructing and generating a first model using the pre-trained VGG16 network and a Markov process; step S3, updating the weights of a policy network in the deep reinforcement learning network by using a deep reinforcement learning network in the first model, and generating a second model; step S4, setting the model structure of the second model and generating a third model; step S5, obtaining multiple random samples from a preset data set; and step S6, training the third model using the multiple random samples. Compared with related technologies, the technical solution of the present invention can effectively improve learning efficiency and reduce computational complexity.

Description

Technical Field

[0001] The present invention relates to the field of deep reinforcement learning technology, and in particular to an active target positioning method, an active target positioning device, and a computer-readable storage medium based on deep reinforcement learning. Background Art

[0002] In recent years, deep reinforcement learning technology has developed rapidly, and object detection in images based on deep reinforcement learning has been increasingly widely used in production and life.

[0003] Related deep reinforcement learning techniques are limited to scenarios where both the action space and the sample space are small and generally discrete. However, more complex tasks that are closer to real-world scenarios often have larger state spaces and continuous action spaces. When the input data is images or sounds, it is often very high-dimensional, making it difficult for related deep reinforcement learning techniques to handle. Deep reinforcement learning combines the high-dimensional inputs that deep learning is good at with reinforcement learning. Object detection in images based on deep reinforcement learning has two drawbacks: First, if the network learns only from continuously occurring empirical samples in the environment, these samples will be highly correlated, resulting in low learning efficiency. Second, while overlapping candidate proposal strategies produce cropped images of each candidate box by computing feature maps for the entire image, experiments have shown that this strategy yields better results. However, due to the loss of spatial resolution, performance for cropped image features is compromised, and this loss is particularly significant when processing a large number of candidate objects. Summary of the Invention

[0004] In view of the above shortcomings of the existing technology, the present invention proposes an active target positioning method, an active target positioning device and a computer-readable storage medium, which can effectively improve learning efficiency and reduce the amount of calculation.

[0005] In order to solve the above technical problems, in a first aspect, an embodiment of the present invention provides an active target localization method applied to a VGG16 network; the method comprises the following steps:

[0006] Step S1: inputting the image to be processed into a pre-trained VGG16 network to extract features of the image to be processed, wherein the pre-trained VGG16 network is a trained VGG16 network obtained after training the VGG16 network;

[0007] Step S2: constructing and generating a first model by using the pre-trained VGG16 network and a Markov process, wherein the first model includes states, actions, and rewards;

[0008] Step S3: updating the weights of the policy network in the deep reinforcement learning network by using the deep reinforcement learning network in the first model, and generating a second model; the second model includes the deep reinforcement learning network, and updating the weights of the policy network to update the weights of the target network in the deep reinforcement learning network to the weights of the policy network at each preset time step;

[0009] Step S4: setting the model structure of the second model and generating a third model; the pre-trained VGG16 network includes 13 convolutional layers, 5 maximum pooling layers, and 3 fully connected layers; the third model structure uses a hierarchical method and a dynamic method to construct the second model, respectively. The hierarchical method is to connect the output of the last maximum pooling layer in the order of arrangement among the 5 layers of the maximum pooling layers with a 24-dimensional action history vector, and the dynamic method is to connect the output of the first fully connected layer in the order of arrangement among the 3 layers of the fully connected layers with a 90-dimensional action history vector.

[0010] Step S5: obtaining multiple random samples through a preset data set;

[0011] Step S6: Use a plurality of the random samples to train the third model.

[0012] Preferably, in step S2, the state includes the current area and the memory vector, and the action includes a moving action and a terminating action.

[0013] Preferably, the intersection-over-union ratio in the Markov process satisfies the following formula (1):

[0014]

[0015] Among them, b is the search box, g is the parameter, and IoU is the intersection over union function;

[0016] The reward function in the Markov process satisfies the following formulas (2) and (3):

[0017] R a (s,s')=sign(IoU(b',g)-IoU(b,g)) (2);

[0018] ;

[0019] Wherein, Ra is the moving action function, Rt is the ending action function, b' is the target box, τ is the trigger threshold, s is the input parameter, s' is the output parameter, and η is the setting value of the dynamic method and the hierarchical method.

[0020] Preferably, η is 3; τ is 0.6.

[0021] Preferably, in step S3, the target value of each time step is calculated by the target network; the preset q value is calculated by the strategy network, and the q value is the numerical value of the learning strategy.

[0022] Preferably, in step S5, the dataset is the PASCAL Visual Object Classes 2012 dataset.

[0023] Preferably, in step S5, the plurality of random samples are images corresponding to objects of the same class in the data set.

[0024] Preferably, in step S6, the image scaling operation in training the third model is achieved by performing a hyperparameter search.

[0025] In a second aspect, an embodiment of the present invention further provides an active target positioning device, comprising a processor and a memory, wherein the processor is configured to read a program in the memory and execute the steps in the above-mentioned active target positioning method provided in an embodiment of the present invention.

[0026] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the steps in the above-mentioned active target positioning method provided in the embodiment of the present invention are implemented.

[0027] Beneficial effects of the present invention: In the present invention, the present invention provides an active target positioning method, an active target positioning device and a computer-readable storage medium. The active target positioning method includes step S1, inputting the image to be processed into a pre-trained VGG16 network; step S2, constructing and generating a first model through the pre-trained VGG16 network and using a Markov process; step S3, updating the weights of the policy network in the deep reinforcement learning network by using a deep reinforcement learning network in the first model, and generating a second model; step S4, setting the model structure of the second model and generating a third model; step S5, obtaining multiple random samples through a preset data set; step S6, using multiple random samples to train the third model. The active target localization method of the present invention treats the target detection problem as a dynamic decision-making process. A Markov process is used to make decisions in step S2. The portion of the image containing the target object is found when training the third model in step S6. Steps S3 to S4 are then implemented to magnify the target object when training the third model in step S6. Multiple random samples are then generated in step S5, and this process is repeated when training the third model in step S6 until a tight bounding box is found around the target object. Therefore, by implementing steps S1 to S6, the active target localization method, active target localization device, and computer-readable storage medium of the present invention can effectively improve learning efficiency while minimizing computational effort. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The present invention will be described in detail below with reference to the accompanying drawings. The above and other aspects of the present invention will become clearer and easier to understand through the detailed description made with reference to the following drawings. In the accompanying drawings:

[0029] Figure 1 is a flowchart of the active target positioning method of the present invention;

[0030] Figure 2 It is a structural schematic diagram of the active target positioning device of the present invention. DETAILED DESCRIPTION

[0031] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0032] The specific embodiments / examples described herein are specific embodiments of the present invention and are used to illustrate the concept of the present invention. They are illustrative and exemplary and should not be construed as limiting the embodiments of the present invention or the scope of the present invention. In addition to the examples described herein, those skilled in the art can also adopt other obvious technical solutions based on the claims and the disclosure of the specification. These technical solutions, including any obvious replacements and modifications of the embodiments described herein, are all within the scope of protection of the present invention.

[0033] The present invention provides an active target positioning method, which is applied to a VGG16 network.

[0034] Please refer to Figure 1 As shown, Figure 1 This is a flowchart of the active target positioning method of the present invention.

[0035] The active target positioning method comprises the following steps:

[0036] Step S1: Input the image to be processed into a pre-trained VGG16 network to extract features of the image to be processed. The pre-trained VGG16 network is a trained VGG16 network obtained after training the VGG16 network.

[0037] Step S2: construct and generate a first model by using the pre-trained VGG16 network and a Markov process.

[0038] The construction includes state, actions and rewards.

[0039] The state includes a current region and a memory vector.

[0040] In this embodiment, one memory vector records the four most recently completed actions for searching for an object and encodes them using one-hot encoding. Since there are six actions (upper left, upper right, lower left, lower right, center, and end), the memory vector has a total of 24 dimensions.

[0041] The actions include movement and terminal.

[0042] In this embodiment, in the hierarchical approach, there are five movement actions corresponding to the five sub-regions in the current bounding box: four representing the four quadrants plus one center region. There is also a terminal operation called trigger, which should be selected when the agent reaches the goal of finding the right border around the target object. In the dynamic approach, there are eight transformation operations used to change the geometry and position of the bounding box, and one trigger operation. These eight transformations can be divided into four categories, respectively modifying the horizontal level, vertical level, scale, and aspect ratio of the border. The following formula (4) is satisfied:

[0043] α w =α(x2-x1)

[0044] α h=α(y2-y1) (4).

[0045] Where α∈[0,1] and x1,y1,x2,y2 are the coordinates of the upper left corner and lower right corner of the currently placed bounding box respectively. Then, according to the type of transformation action, α w and α h Add or subtract the current bounding box coordinates.

[0046] The intersection-over-union ratio in the Markov process satisfies the following formula (1):

[0047]

[0048] Among them, b is the search box, g is the parameter, and IoU is the intersection over union function;

[0049] The reward function in the Markov process satisfies the following formulas (2) and (3):

[0050] R a (s,s')=sign(IoU(b',g)-IoU(b,g))(2).

[0051] .

[0052] Where Ra is the movement action function, Rt is the termination action function, b' is the target box, τ is the trigger threshold, s is the input parameter, s' is the output parameter, and η is the setting value of the dynamic method and the hierarchical method. Formula (2) represents the reward equation as the difference between the IoU of the search box and the target box before and after the action (from the search box b to the target box b').

[0053] Formula (3) represents that the feedback mechanism is different because the terminal operation does not change the geometry of the bounding box. Therefore, the feedback is defined based on whether the current IoU exceeds a pre-set threshold (τ).

[0054] In this example, η is 3 and τ is 0.6. If the current IOU is greater than the threshold, the terminal reward is set to 3, otherwise it is set to -3. The hierarchical method takes 0.5.

[0055] Step S3: Update the weights of the policy network in the deep reinforcement learning network by adopting the deep reinforcement learning network in the first model, and generate a second model.

[0056] The second model includes the deep reinforcement learning network.

[0057] The weight of the policy network is updated so that the weight in the target network in the deep reinforcement learning network is updated to the weight of the policy network at each preset time step.

[0058] The target value of each time step is calculated by the target network; the preset q value is calculated by the strategy network, and the q value is the numerical value of the learning strategy.

[0059] One problem with DQN, or reinforcement learning algorithms in general, is overestimation of Q-values. In other words, as our Q-values ​​get closer to their targets, the targets continue to move because we use the same network to calculate both values. To address this, we implement two independent networks: one for estimating the target value at each step, called the target network, and the other for learning the policy (Q-value), called the policy network. The target network is a clone of the policy network. Its weights remain consistent with those of the original policy network, and at regular time steps, the weights in the target network are updated to the new weights of the policy network.

[0060] Step S4: Set the model structure of the second model and generate a third model.

[0061] The pre-trained VGG16 network includes 13 convolutional layers, 5 maximum pooling layers, and 3 fully connected layers.

[0062] The third model structure uses a hierarchical method and a dynamic method to construct the second model.

[0063] The hierarchical method is to connect the output of the last maximum pooling layer in the arrangement order of the 5 layers of the maximum pooling layers with the 24-dimensional action history vector to form a state representation of the state.

[0064] The dynamic method is to connect the output of the first fully connected layer in the arrangement order of the three fully connected layers with a 90-dimensional action history vector to form a state representation of the state.

[0065] The final array in the third model structure is fed into a DQN consisting of three fully connected layers. The first two fully connected layers consist of 1024 units, and the last fully connected layer contains the q-value of each action. Among them, 6 units are allocated in the hierarchical method and 9 units are allocated in the dynamic method. It is worth mentioning that in the DQN structure of the related art, the output layer does not have any nonlinear activation function because the third model needs to obtain the untransformed original q-value from the deep reinforcement learning network. In the implementation of the third model, the output layer in the third model uses a linear activation function.

[0066] Step S5: Obtain multiple random samples through a preset data set.

[0067] In this example, the dataset is the PASCAL Visual Object Classes 2012 dataset. This dataset consists of images from 20 object categories, such as humans, birds, cats, cows, and dogs. This dataset contains approximately 12,000 images, providing ground truth boxes for model training and testing.

[0068] In this embodiment, the plurality of random samples are images corresponding to objects of the same category in the data set.

[0069] Step S6: Use a plurality of the random samples to train the third model.

[0070] The DQN in the third model accepts the state of a given environment as input. For each given state, the deep reinforcement learning network can output an estimated value of q for each state's action. The goal of the deep reinforcement learning network is to optimize the q function derived from the Bellman equation, satisfying the following formula:

[0071]

[0072] The loss of the deep reinforcement learning network is calculated by comparing the output Q-value with the true optimal Q-value. The goal of the deep reinforcement learning network is to minimize this loss. After calculating the loss, the weights of the deep reinforcement learning network are updated through stochastic gradient descent and backpropagation. This process is repeated for all states of the environment until the loss is small enough and an approximate optimal Q-function is obtained. It satisfies the following formula:

[0073]

[0074] Training the third model requires a special setting, which is a sample pool (experience replay). The experience of training the third model is stored in the sample pool and each time step, defined as: et = (current state, action, feedback, next state). If the deep reinforcement learning network only learns from experience samples that occur continuously in the environment, then these samples will be highly correlated, resulting in low learning efficiency. The DQN is then trained using random samples in the replay memory to break the correlation between consecutive samples. Using the estimated q-value of the deep reinforcement learning network, we can find an optimal object region proposal strategy to locate objects of a specific category in the image.

[0075] In this embodiment, the image scaling operation in training the third model is achieved by performing a hyperparameter search.

[0076] By implementing the above-mentioned steps S1 to S6, the active target localization method of the present invention treats the target detection problem as a dynamic decision-making process. In step S2, a Markov process is used to make a decision. When training the third model in step S6, the portion of the image containing the target object is found. Then, by implementing steps S3 to S4, the target object is magnified when training the third model in step S6. Then, through multiple random samples in step S5, this process is repeated when training the third model in step S6 until a tight bounding box is found around the target object. Specifically, the active target localization method of the present invention solves the problem that the network learns from continuous experience samples in the environment, and the samples are highly correlated, resulting in low learning efficiency. Then, the random samples in the replay memory in the third model are used to train the deep reinforcement learning network (Deep Q Network, DQN) to break the correlation between continuous samples. The active target localization method of the present invention solves to some extent the problem of performance loss of cropped image features due to loss of spatial resolution. The number of candidate boxes generated by implementing the above-mentioned steps S1 to S6 is greatly reduced, allowing for feature extraction for each position without sharing convolution calculations between regions. A hierarchical representation is introduced to guide the third model through the image from top to bottom (zoom in), and how the design of the hierarchical structure affects the detection performance and the number of visited areas is explored. The active target localization method of the present invention extracts features for each area instead of reusing feature maps of multiple locations. The scheme shows the gain of region-specific features and believes that since the number of regions considered by the third model is very small, the computational overhead is very small. The third model represents the image through a hierarchical structure, thereby taking into account the connection between blocks. At the same time, no post-processing of non-maximum suppression is required, which reduces the detection area.

[0077] The present invention also provides an active target positioning device 1000. Figure 2 As shown, Figure 2 FIG. 1 is a structural diagram of an active target positioning device 1000 of the present invention.

[0078] The active target positioning device 1000 includes a processor 1001, a memory 1002, a network interface 1003, and a computer program stored in the memory 1002 and executable by the processor 1001. The processor 1001 is configured to read the program in the memory 1002. When the processor 1001 executes the computer program, the steps of the active target positioning method provided in the embodiment are implemented. That is, the processor 1001 executes the steps of the active target positioning method.

[0079] Specifically, the processor 1001 is configured to perform the following steps:

[0080] Step S1: Input the image to be processed into a pre-trained VGG16 network to extract features of the image to be processed. The pre-trained VGG16 network is a trained VGG16 network obtained after training the VGG16 network.

[0081] Step S2: construct and generate a first model using the pre-trained VGG16 network and a Markov process, wherein the first model includes states, actions, and rewards.

[0082] Step S3: Using a deep reinforcement learning network in the first model to update the weights of the policy network in the deep reinforcement learning network, and generating a second model. The second model includes the deep reinforcement learning network, and the weights of the policy network are updated so that the weights in the target network in the deep reinforcement learning network are updated to the weights of the policy network at each predetermined time step.

[0083] Step S4: Set the model structure of the second model and generate a third model; the pre-trained VGG16 network includes 13 convolutional layers, 5 maximum pooling layers, and 3 fully connected layers. The third model structure uses a hierarchical method and a dynamic method to construct the second model. The hierarchical method is to connect the output of the last maximum pooling layer in the five layers of the maximum pooling layer with a 24-dimensional action history vector, and the dynamic method is to connect the output of the first fully connected layer in the three layers of the fully connected layer with a 90-dimensional action history vector.

[0084] Step S5: Obtain multiple random samples through a preset data set.

[0085] Step S6: Use a plurality of the random samples to train the third model.

[0086] The active target positioning device 1000 provided in the embodiment of the present invention can implement various implementations and corresponding beneficial effects in the embodiment of the active target positioning method, which will not be described again here to avoid repetition.

[0087] It should be pointed out that Figure 2Only components 1001-1003 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art will understand that the active target positioning device 1000 here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0088] The memory 1002 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, the memory 1002 may be an internal storage unit of the active target positioning device 1000, such as a hard disk or memory of the active target positioning device 1000. In other embodiments, the memory 1002 may also be an external storage device of the active target positioning device 1000, such as a plug-in hard disk equipped on the active target positioning device 1000, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Of course, the memory 1002 may also include both the internal storage unit of the active target positioning device 1000 and its external storage device. In this embodiment, the memory 1002 is generally used to store the operating system and various application software installed on the active target positioning device 1000, such as the program code of the active target positioning method of the active target positioning device 1000. In addition, the memory 1002 can also be used to temporarily store various data that has been output or is about to be output.

[0089] In some embodiments, the processor 1001 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 1001 is generally used to control the overall operation of the active target positioning device 1000. In this embodiment, the processor 1001 is used to execute program code or process data stored in the memory 1002, such as executing program code for the active target positioning method of the active target positioning device 1000.

[0090] The network interface 1003 may include a wireless network interface or a wired network interface. The network interface 1003 is generally used to establish a communication connection between the active target positioning device 1000 and other electronic devices.

[0091] The present invention also provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions, and when the program instructions are executed by the processor 1001, the steps in the active target positioning method are implemented.

[0092] Those skilled in the art will appreciate that all or part of the processes in the active target positioning method of the active target positioning device 1000 described in the embodiments can be performed by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the methods. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0093] The embodiments mentioned in the embodiments of the present invention are for the sake of convenience. The above disclosure is only a preferred embodiment of the present invention and certainly cannot be used to limit the scope of the rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

[0094] In the present invention, an active target positioning method, an active target positioning device and a computer-readable storage medium are provided. The active target positioning method includes step S1, inputting the image to be processed into a pre-trained VGG16 network; step S2, constructing and generating a first model through the pre-trained VGG16 network and using a Markov process; step S3, updating the weights of the policy network in the deep reinforcement learning network by using a deep reinforcement learning network in the first model, and generating a second model; step S4, setting the model structure of the second model and generating a third model; step S5, obtaining multiple random samples through a preset data set; step S6, using multiple random samples to train the third model. The active target localization method treats the target detection problem as a dynamic decision-making process. Step S2 uses a Markov process to make decisions. When training the third model in step S6, the portion of the image containing the target object is found. Steps S3 to S4 are then implemented to magnify the target object when training the third model in step S6. Multiple random samples are then generated in step S5, and this process is repeated until a tight bounding box is found around the target object when training the third model in step S6. Therefore, by implementing steps S1 to S6, the active target localization method, active target localization device, and computer-readable storage medium of the present invention can effectively improve learning efficiency and reduce computational complexity.

[0095] The above description is only an embodiment of the present invention. It should be pointed out that those skilled in the art can make improvements without departing from the creative concept of the present invention, but these improvements all fall within the scope of protection of the present invention.

Claims

1. An active target localization method, which is applied to the VGG16 network; characterized in that: The method comprises the following steps: Step S1: inputting the image to be processed into a pre-trained VGG16 network to extract features of the image to be processed, wherein the pre-trained VGG16 network is a trained VGG16 network obtained after training the VGG16 network; Step S2: constructing and generating a first model by using the pre-trained VGG16 network and a Markov process, wherein the first model includes states, actions, and rewards; Step S3: updating the weights of the policy network in the deep reinforcement learning network by using the deep reinforcement learning network in the first model, and generating a second model; the second model includes the deep reinforcement learning network, and updating the weights of the policy network by updating the weights of the target network in the deep reinforcement learning network to the weights of the policy network at every preset time step; Step S4: setting the model structure of the second model and generating a third model; the pre-trained VGG16 network includes 13 convolutional layers, 5 maximum pooling layers, and 3 fully connected layers; the third model structure uses a hierarchical method and a dynamic method to construct the second model, respectively. The hierarchical method is to connect the output of the last maximum pooling layer in the order of arrangement among the 5 layers of the maximum pooling layers with a 24-dimensional action history vector, and the dynamic method is to connect the output of the first fully connected layer in the order of arrangement among the 3 layers of the fully connected layers with a 90-dimensional action history vector. Step S5: obtaining multiple random samples through a preset data set; Step S6: Use a plurality of the random samples to train the third model.

2. The active target positioning method according to claim 1, characterized in that: In step S2, the state includes the current area and the memory vector, and the action includes a moving action and a terminal action.

3. The active target positioning method according to claim 2, characterized in that: The intersection-over-union ratio in the Markov process satisfies the following formula (1): (1)； Among them, b is the search box, g is the parameter, I oU is the intersection-over-union function; The reward function in the Markov process satisfies the following formulas (2) and (3): (2)； (3) ； Among them, Ra is the moving action function, Rt is the ending action function, b' is the target frame, is the trigger threshold, s is the input parameter, s' is the output parameter, and η is the setting value of the dynamic method and the hierarchical method.

4. The active target positioning method according to claim 3, characterized in that: η is 3; is 0.

6.

5. The active target positioning method according to claim 1, characterized in that: In step S3, the target value of each time step is calculated by the target network; the preset q value is calculated by the strategy network, and the q value is the value of the learning strategy.

6. The active target positioning method according to claim 1, characterized in that: In step S5, the dataset is the PASCAL Visual Object Classes 2012 dataset.

7. The active target positioning method according to claim 1, characterized in that: In step S5, the plurality of random samples are all images corresponding to objects of the same class in the data set.

8. The active target positioning method according to claim 1, characterized in that: In step S6, the image scaling operation in the training of the third model is achieved by performing a hyperparameter search.

9. An active target positioning device, characterized in that: The method comprises a processor and a memory, wherein the processor is configured to read a program in the memory and execute the steps of the active target positioning method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the steps in the active target positioning method according to any one of claims 1 to 8 are implemented.

Images