Reinforcement learning based salient object ranking method

Abstract

The application discloses a salient target ranking method based on reinforcement learning, constructs an environment and an actor in an offline stage, and trains the actor and the environment through a salient target ranking data set to obtain a salient target ranking result in a scene. The application uses dynamic interaction with the environment to learn an optimal ranking strategy, so that the model learns to flexibly adjust according to factors such as context information, mutual relationship between targets and reward signals, thereby simulating the dynamic transfer process of human visual attention, generating more accurate and practical salient target ranking results, and providing more practical technical support for the application of intelligent auxiliary tools for visually impaired people, image description generation, human-computer interaction, auxiliary decision-making, intelligent monitoring, automatic driving, augmented reality and the like.

Description

Technical Field

[0001] This invention relates to a technique in the field of image processing, specifically a salient target ranking method based on reinforcement learning. Background Technology

[0002] Existing deep learning-based salient object ranking techniques are usually transformed into classification or regression problems, achieving prediction through static feature extraction. They lack dynamic adaptability and struggle to capture the dynamic shifts in human visual attention mechanisms based on context and the relationship between objects. As a result, they cannot accurately reflect the natural flow of human vision in ranking tasks, affecting the accuracy and practicality of the results. Summary of the Invention

[0003] This invention addresses the limitation of existing technologies in observing the dynamic movement order between salient targets by proposing a reinforcement learning-based salient target ranking method. This method utilizes dynamic interaction with the environment to learn the optimal ranking strategy, enabling the model to flexibly adjust based on contextual information, inter-target relationships, and reward signals. This simulates the dynamic shift process of human visual attention, generating more accurate and practical salient target ranking results. This provides more practical technical support for applications in fields such as intelligent assistive tools for visually impaired patients, image description generation, human-computer interaction, decision support, intelligent monitoring, autonomous driving, and augmented reality.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a salient object ranking method based on reinforcement learning. In the offline stage, an environment and an agent are constructed, and the interaction between the agent and the environment is trained using a salient object ranking dataset to obtain the ranking results of salient objects in the scene.

[0006] The environment includes an Environment Candidate Generator (ECG), an Action Executor, and a Memory Unit. The ECG segments all target instances in the original image and extracts their corresponding features, then filters them to obtain salient candidate target instances and their feature sets. The Memory Unit records and maintains the initial state using the feature set. The agent, based on the initial state, sequentially performs spatial context information feature fusion and temporal context information fusion to predict the cumulative score of the executed action. The Action Executor judges the overall performance of the currently selected target instance based on the cumulative score, and then rewards or punishes the action according to the set reward strategy to update the agent's parameters. Simultaneously, based on the relationship between the agent's selection result and the processed target instances, it records the selected target instances and their order, and updates the state in the Memory Unit, i.e., the remaining candidate target instances and their feature sets.

[0007] The Environment Candidate Generator (ECG) extracts and filters salient candidate target instances and their feature sets from the original image, including: a target instance segmentation module, a feature extraction module, and a candidate target filtering module. Specifically: the target instance segmentation module generates masks and bounding boxes for all target instances based on the original image and outputs them along with the original image to the feature extraction module to obtain the features of all target instances; the candidate target filtering module filters salient candidate target instances and their feature sets from the set of features of all target instances based on the bounding boxes of the target instances.

[0008] The intelligent agent includes: a spatial attention module (SAM), a temporal attention module (TAM), and a scorehead. The spatial attention module integrates information about surrounding target instances, i.e., spatial context information, into the features of the target instances. The temporal attention module calculates the temporal interaction between salient targets, i.e., the degree of salience of the target instances noticed in the previous time step, which affects the target instances noticed in the next time step. The scorehead predicts the value of each action, i.e., the cumulative score predicted by the scorehead.

[0009] The integration mentioned refers to: through the structure of self-attention, remembering This is a set of features for candidate target instances that incorporates spatial context information. For target instances containing spatial context information Features The number of candidate target instances. Global features. This is the P5 feature map of the original image, denoted as dimensionality. The set of salient candidate target instance features is The set of candidate target instance features with fused spatial context information is calculated as follows. ,in: , , , , The three values ​​represent the query vector, key vector, and value vector in the attention mechanism, respectively. , and These are trainable parameters whose function is to convert the input vector into Query, Key, and Value vectors; This is the spatial attention matrix, representing the attention distribution in space; T is the transpose operation; It is acting on The parameters that can be trained.

[0010] The aforementioned calculation of the temporal interaction between saliency targets refers to: calculating the feature set of candidate target instances that incorporates temporal context information. ,in: For the target instance noticed at the current time step Features Let be the number of candidate target instances. Features of the target instance noticed in the previous time step, For target instances containing spatial context information Features , These are trainable parameters.

[0011] The value of predicting the selection of each action refers to: based on the set The action of selecting a specific target instance and Number of target instances Calculate in state Select action And continue to use the current strategy. The cumulative score from multiple interactions with the environment until the end of the round. , , and These are trainable parameters. It is a global feature Features after linear layer mapping.

[0012] The aforementioned updating of agent parameters refers to updating the parameters of each module network of the agent using a temporal difference algorithm, based on two agents with identical structures but different parameters simultaneously participating in training and calculating the loss. Specifically, the parameter update frequency of the agent serving as the target model is much lower than that of the learning model. Let the strategy of the target model be denoted as... Then the target model predicts in At any given moment, the cumulative score is calculated based on the interaction between the target model's strategy and the environment until the end of the round. , The cumulative score is calculated by continuously interacting with the environment according to the strategy of the learning model agent until the end of the entire round. The environment's influence The action selected by the learning model in real time The rating is The goal of model training is to make the model training process so that the model achieves the desired results. and The value should be as close as possible.

[0013] Technical effect

[0014] This invention, by constructing a dynamically adjusted module incorporating time-varying features and combining it with a real-time monitoring and feedback optimization mechanism for data distribution, can capture the changes of salient targets under the influence of spatial and temporal contexts, extract more representative and refined features, and improve model performance. Compared with existing technologies, this invention solves the problems of static sorting and difficulty in capturing context and relationships between targets in traditional methods. The generated results are more consistent with the changing patterns in real-world applications and can be widely used in fields such as intelligent decision-making and real-time monitoring, providing a new approach to addressing the limitations of traditional modeling methods in dynamic environments. By dynamically adjusting the modeling strategy and utilizing the interaction between the environment and the agent to simulate the dynamic transfer process of human visual attention, this invention improves sorting accuracy while broadening the application scope of the model, providing a solution for intelligent modeling and efficient analysis of complex systems. Attached Figure Description

[0015] Figure 1 This is a flowchart of the present invention;

[0016] Figure 2 This is a network diagram for an example embodiment;

[0017] Figure 3 (a) is a memory unit and Figure 3 (b) Flowchart of the motion actuator;

[0018] Figure 4 This is a flowchart of an implementation example;

[0019] Figure 5 This is a schematic diagram illustrating the effect of an example. Detailed Implementation

[0020] like Figure 1 and Figure 2 As shown in the figure, this embodiment relates to a salient target ranking method based on reinforcement learning. In the offline stage, an environment including an environment candidate generator, an action executor, and a memory unit is constructed, and an agent including a spatial attention module, a temporal attention module, and a scoring head is constructed. The agent is trained to interact with the environment through a salient target ranking dataset to obtain the ranking result of salient targets in the scene.

[0021] like Figure 2As shown, the environment candidate generator includes a target instance segmentation module, a feature extraction module, and a candidate target selection module. Specifically, the target instance segmentation module uses the CenterMask model and generates target instance masks and bounding boxes using the model's mask head and bounding box head. The feature extraction module uses the result of a bitwise AND operation between the target instance's mask and the original image as input. It then uses the backbone network of the CenterMask model combined with a feature pyramid network to extract multi-layer, multi-scale feature maps of all target instances. The fifth-layer feature map (P5) of the feature pyramid is used as the output target instance features and scene features, ultimately forming a set of all target instances and their features. The candidate target selection module selects salient candidate target instances and their feature sets from the set of all target instances and their features. Specifically, it first calculates the Intersection over Union (IoU) based on the ground truth and predicted target instance bounding boxes, and then selects truly salient targets based on the IoU.

[0022] The filtering process involves setting the predicted target instance's bounding box to a salient target if the intersection-union ratio (IU) between the predicted target instance and the ground truth annotation is greater than 0.5, with the salientness being the same as the ground truth annotation. If the IU is less than 0.5, the salientness is set to the lowest level, i.e., not salient.

[0023] like Figure 2 As shown, the memory unit records and maintains the state in the environment, transmits the current state to the agent, and outputs the ranking result of salient targets.

[0024] like Figure 3 As shown in (a), the memory unit constructs a mapping from the set of target instance identifiers to the set of target instance related information, i.e., a dictionary data structure. ,in: For the first A unique identifier for each target instance, specifically a numeric number. It is a set of relevant information corresponding to the target instance. For the total number of target instances, the th The target instance corresponding to , For the first When a target instance is selected. ; For the first When a target instance has been selected, record its selection order, which is a non-negative integer. Let its value be... ,satisfy ,in And for all selected target instances in each round, its The value is unique and is assigned in ascending order according to the selection sequence. The dictionary data structure stores the feature set of candidate target instances. Secondly, throughout the operation of the saliency ranking network, as the interactions progress, the dictionary data structure... Information related to each target instance in The system records the order in which target instances are selected based on their selection status. Meanwhile, the remaining candidate target instances and their feature sets are sent to the agent as new states for the next round of interaction, until the final output of the saliency target ranking results is completed at the end of the round.

[0025] like Figure 2 As shown, the action executor updates the state of the memory unit based on the result of the agent's selection and rewards or punishes the agent's actions based on the real significance of the target instance.

[0026] like Figure 3 As shown in (b), when the agent selects the first When the target instance is the most prominent target among the remaining target instances, the action executor updates the dictionary data structure of the memory unit. When the corresponding target instance is selected, from Modified to And record the order in which the target was selected: At the same time, the remaining candidate target instances and their feature sets are updated into memory units. The new status: , For update operations. Furthermore, the action executor, based on the true salience of the salient target, rewards or penalizes the agent for the score of performing the current action by setting a reward strategy, and feeds this score back to the agent. The reward strategy adopted is: if the agent selects the most salient of the remaining target instances, a reward is given; otherwise, a penalty is given (i.e., no cumulative score is added in this round). Specifically, let the target instance... The true significance is The value is in Interval; when the agent selects the first interval; When the target instance is the most prominent target among the remaining target instances, let its initial score be... The score after processing by the motion actuator is: : The reward function Set as , As the reward coefficient, This represents the true significance level.

[0027] like Figure 2As shown, the interaction specifically involves: the environment providing salient candidate target instances and their feature sets (states); the agent selecting the target with the highest salientity from the set (action); the environment recording the agent's selection and removing that target from the set (new state); this constitutes one interaction; the process is repeated multiple times until all candidate target instances are removed from the set, constituting one round. During the training phase, a round ends after the agent completes the sorting of all candidate target instances. During the inference phase, the agent's action space is expanded with active stop actions, providing the agent with the opportunity to terminate the interaction early. When the agent determines that choosing any target will only result in a negative score, it can choose to end the sorting, i.e., end the round.

[0028] Through specific practical experiments, network deployment and model training were completed on a computer device with an Intel 9700k CPU, an NVIDIA RTX 3090 Ti graphics card, and 64GB of RAM. For example... Figure 4 As shown, this embodiment presents a saliency target ranking method based on reinforcement learning, which includes:

[0029] Step 1: Prepare the dataset, which includes:

[0030] 1.1 Download the open-source salient object ranking dataset ASSR. Each original RGB scene image in this dataset has a corresponding salient ranking ground truth image. This ground truth image is a grayscale image. The dataset's annotations consist of two types of files. One type is the salient object annotation file, containing information such as the location, category, and mask of salient objects in each image; the other type is the salientity score file, containing the salientity score of each salient object in the scene. Here, the salientity score is a floating-point value ranging from [0, 1].

[0031] 1.2 The training set, validation set, and test set are divided into three parts according to a ratio of 7:1:2.

[0032] 1.3 Before training begins, the dataset needs to be preprocessed. To accelerate the training process, target instances and their features are pre-extracted from the original images and stored locally, thus reducing redundant feature extraction steps. After using the CenterMask target instance segmentation model to predict the target locations in the images and their corresponding masks, these masks are applied to the original images to obtain target instances. Then, the backbone network and feature pyramid network in CenterMask are used to extract multi-layer, multi-scale features of the target instances. Subsequently, these target locations, the set of all target instances and their features, are stored locally and used as input to the execution components of the model.

[0033] Step 2, model training, specifically includes:

[0034] 2.1 The loss function used is the mean squared error (MSE), specifically:

[0035] 2.2 Hyperparameter settings are as follows: The threshold for the candidate target selection module in the environment is set to 0.5. The reward policy for the action executor in the environment is set as follows: if the agent selects the most significant target, points are added; if a non-significant target is selected, points are deducted; otherwise, no points are added or deducted. The reward coefficient is set to 1. The number of update iterations is set to 20,000. During the training phase, the probability of randomly selecting objects is set to 0.9, the learning rate is set to 1e-6, and the batch size is set to 4. The agent is trained for 11,000 epochs using the training set obtained in step 1, and the model performance is evaluated on the validation set. The model structure or hyperparameters are adjusted based on the validation results to obtain the best-performing model.

[0036] Step 3: Use the Mean Absolute Error (MAE) and f1-Sor metrics to evaluate model performance.

[0037] The MAE metric is calculated as follows: First, a saliency map is drawn based on the saliency of the targets. Targets with higher saliency are brighter in the map. Then, the pixel-by-pixel difference between the normalized predicted saliency map and the actual saliency map is calculated, and the average value is then taken. MAE measures the pixel-level difference between the predicted saliency ranking map and the true value. MAE considers not only the accuracy of ranking the top N most important objects but also the model's recall and error rate.

[0038] The f1-Sor metric is calculated as follows: First, calculate the f1 value and the Sor value separately, then multiply them to obtain the f1-Sor score. Here, Sor is the Pearson correlation coefficient. The f1 value is calculated as follows: First, calculate the recall and precision of the salient target, then calculate: .

[0039] Table 1 shows the performance evaluation results of the model test set. The specific ranking results of the generated saliency targets are as follows: Figure 5 As shown.

[0040] Table 1. Performance Evaluation of the Model Test Set

[0041]

[0042] As shown in Table 1, on the ASSR dataset, the trained model achieves a salient object ranking accuracy (f1-SOR) of 0.788, reaching the current best level (the short line indicates a lack of publicly available data, making it impossible to calculate the metric). Similarly, on the MAE metric, the trained salient object ranking model also stands out with a performance of 0.074, further validating its ability to dynamically calculate the salient ranking process and accurately capture and rank salient objects in images.

[0043] Table 2 shows the testing and evaluation of key aspects for improving model performance. The baseline model involves directly inputting target features into a linear layer, which is directly trained using reinforcement learning. Building upon this, integrating either the SAM or TAM module improves the model's performance on the F1-Sor and MAE metrics. Combining both SAM and TAM modules simultaneously achieves the best overall performance, indicating a synergistic effect when using these two modules.

[0044] Table 2. Test results of key aspects of model performance improvement

[0045]

[0046] Compared with existing technologies, the core advantage of this invention lies in its dynamic modeling process, overcoming the limitations of traditional static modeling and effectively improving the performance of ranking salient targets. By constructing the model as a dynamic and continuously optimizing process, this method can adapt to changes in the data environment in real time, thereby significantly improving the model's flexibility and applicability. In practical application scenarios, data often exhibits non-static characteristics. Traditional methods typically achieve prediction through static feature extraction, lacking dynamic adaptability and failing to capture the dynamic shifts in human visual attention mechanisms based on context and the relationships between targets. This results in the inability to accurately reflect the natural flow of human vision in ranking tasks, affecting the accuracy and practicality of the results. This method, however, dynamically adjusts the modeling strategy, utilizing the interaction between the environment and the agent to simulate the dynamic shifts in human visual attention. Through the TAM and SAM modules in the agent, it enhances the model's contextual understanding and target relationship capture capabilities, achieving more accurate capture of the temporal characteristics and potential change patterns of the data, effectively solving this problem. Overall, this method, through a dynamic modeling process, overcomes the limitations of static modeling, providing a flexible, efficient, and reliable solution for complex data modeling, with broad application prospects and practical value.

[0047] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A salient object ranking method based on reinforcement learning, characterized in that, The environment and the agent are constructed in an offline stage, and the interaction between the agent and the environment is trained through a saliency target ranking dataset to obtain a result of saliency target ranking in a scene. The environment comprises an environment candidate generator (ECG), an action executor and a memory unit, wherein: the ECG divides all target instances in an original image and extracts corresponding target instance features, and then obtains salient candidate target instances and their feature sets through screening; the memory unit records and maintains the state with the feature sets as initial states, the agent sequentially performs spatial context information feature fusion and temporal context information fusion according to the initial state, and then predicts the cumulative score of the executed action; the action executor judges the overall performance of the current selected target instance according to the cumulative score, and then rewards or punishes the action according to the set reward strategy to update the parameters of the agent, and records the selected target instances and their order according to the relationship between the selected result of the agent and the processed target instances, and updates the state in the memory unit, i.e. the remaining candidate target instances and their feature sets. The agent comprises a spatial attention module (SAM), a temporal attention module (TAM) and a score head (ScoreHead), wherein: the spatial attention module integrates the information of surrounding target instances, i.e. spatial context information, into the target instance features, the temporal attention module calculates the time interaction between salient target instances, i.e. the degree of influence of the target instance noticed at the previous time step on the saliency of the target instance noticed at the next time step, and the score head predicts the value of each action (Action), i.e. the cumulative score predicted by the score head.

2. The reinforcement learning based salient target ranking method according to claim 1, characterized in that, The environment candidate generator (ECG) extracts and screens salient candidate target instances and their feature sets from an original image, comprising a target instance segmentation module, a feature extraction module and a candidate target screening module, wherein: the target instance segmentation module generates masks (Mask) and target instance bounding boxes (BoundingBox) of all target instances according to the original image and outputs them to the feature extraction module together with the original image to obtain the features of all target instances; the candidate target screening module screens salient candidate target instances and their feature sets from the set of all target instance features according to the target instance bounding boxes (BoundingBox).

3. The reinforcement learning based salient target ranking method according to claim 1, characterized in that, The integration refers to: through the structure of self-attention, record The selected target instance feature set fused with spatial context information, The feature of the target instance containing spatial context information, The number of selected target instances, global features P5 feature map of the original image, record the dimension of the feature map The selected target instance feature set with significant is And the selected target instance feature set fused with spatial context information is calculated in the following way Wherein: , , , , Three values represent the query vector (Query), key vector (Key) and value vector (Value) in the attention mechanism respectively; , And Are trainable parameters, which are used to convert input vectors into Query, Key and Value vectors; Is the spatial attention matrix, which is the attention distribution on the space; T is the transpose operation; Is a trainable parameter acting on .

4. The reinforcement learning based salient target ranking method of claim 1, wherein, The calculating time interaction between the saliency targets refers to: calculating a candidate target instance feature set fused with time context information Wherein, is a feature of a target instance noticed at a current time step, is a number of candidate target instances, let is a feature of a target instance noticed at a previous time step, is a feature of a target instance containing spatial context information, , is a trainable parameter.​​ 5. The reinforcement learning based salient object ranking method of claim 1, wherein, The predicted selection of the value of each action refers to: according to the number of target instances in the set selected from the set of target instances , , the number of target instances in the set , the number of target instances in the set and continue to use the current strategy interact with the environment multiple times until the cumulative score at the end of the round , , and are trainable parameters, is a global feature feature after linear layer mapping.

6. The reinforcement learning based salient target ranking method according to claim 1, characterized in that, The environment candidate generator comprises a target instance segmentation module, a feature extraction module and a candidate target screening module, wherein: the target instance segmentation module adopts a CenterMask model, and generates a target instance mask and a target instance bounding box through a mask head (Mask head) and a bounding box head (Bounding Box head) of the model; the feature extraction module uses the result of bitwise and (BitwiseAnd) calculation of the Mask of the target instance and an original image as input, then uses a backbone network of the CenterMask model and combines with a feature pyramid network to extract multi-layer and multi-scale feature maps of all target instances, and uses the fifth layer feature map (P5) in the feature pyramid as output target instance features and scene features, finally forms a set of all target instances and their features; the candidate target screening module screens out a set of target instances and their features with significant from the set of all target instances and their features, specifically: first, calculate the intersection over union (IoU) according to the true value and the predicted target instance bounding box, then screen out the real significant target according to the intersection over union.

7. The reinforcement learning based salient target ranking method according to claim 1, characterized in that, The memory unit records and maintains the state of the environment and transmits the current state to the agent, while also outputting the final saliency target ranking result, that is, recording the candidate and selected states of target instances. Specifically, it constructs a mapping from the set of target instance identifiers to the set of target instance related information, i.e., a dictionary data structure. ,in: For the first A unique identifier for each target instance, specifically a numeric number. It is a set of relevant information corresponding to the target instance. For the total number of target instances, the th The target instance corresponding to , For the first When a target instance is selected. ; For the first When a target instance has been selected, record its selection order, which is a non-negative integer, and let its value be... ,satisfy ,in And for all selected target instances in each round, its The values ​​are unique and are assigned sequentially in ascending order of selection. The feature set of candidate target instances is stored. Secondly, during the operation of the saliency ranking network, the memory units are updated as interactions progress. Information related to each target instance in The system records the order in which target instances are selected based on their selection status. Meanwhile, the remaining candidate target instances and their feature sets are sent to the agent as new states for the next round of interaction, until the final output of the saliency target ranking results is completed at the end of the round.

8. The reinforcement learning based salient target ranking method according to claim 1, characterized in that, The action executor updates the state of the memory unit according to the result selected by the agent, and rewards or punishes the action of the agent according to the real saliency of the target instance, specifically: when the agent selects the first target instance as the most salient target in the remaining target instances, the action executor updates the selected state of the corresponding target instance in the memory unit from to , and records the order of the target being selected: At the same time, the remaining target instances to be selected and their feature sets are updated to the new state of the memory unit: , The update operation, in addition, the action executor will reward or punish the score of the agent performing the current action according to the real saliency of the salient target, and feed back the score to the agent, and the reward strategy adopted is: if the agent selects the most salient target in the remaining target instances, it will be rewarded, otherwise it will be punished (i.e. no cumulative points for this round), specifically: let the real saliency of the target instance be , the value is in the interval ; when the agent selects the first target instance as the most salient target in the remaining target instances, let its initial score be , and the score after processing by the action executor is : , where the reward function is set to , is the reward coefficient, is the real saliency value.​​ 9. The reinforcement learning based salient object ranking method of claim 7, wherein, The interaction of the agent and the environment is as follows: the environment currently provides a set of target instances and their features with significant, i.e. the state, the agent selects the target with the highest significant degree from the set, i.e. the action, the environment records the selection of the agent and removes the target from the set, i.e. the new state, and completes one interaction; the interaction is repeated for several times until all target instances are removed from the set for one round; In the training phase, when the agent completes the sorting of all target instances, one round ends; In the reasoning phase, the action space of the agent increases the initiative to stop action, which provides the agent with the opportunity to terminate the interaction in advance, when the agent judges that no matter which target is selected, only negative scores can be obtained, the agent selects the end of the sorting, i.e. the end of the round.

Images