An adaptive control method and system based on visual reinforcement learning

Abstract

The application discloses a kind of self-adaptive control method and system based on visual reinforcement learning, belong to artificial intelligence technical field.The method includes: constructing control model;Training data are constructed;Control model is trained on training data, and is combined with online encoder, online mapping layer and value network, to calculate redundancy reduction loss and reinforcement learning loss;Wherein, the redundancy reduction loss is obtained by minimizing the correlation between different feature dimensions and maximizing the consistency of corresponding dimensions under cross view of the same image;Based on redundancy reduction loss and reinforcement learning loss, the parameters of control model, online encoder, online mapping layer and value network are updated to obtain the trained control model;Based on the control information of the target image obtained by the trained control model.The application improves the control precision and robustness of robot in dynamic interference environment.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to an adaptive control method and system based on visual reinforcement learning. Background Technology

[0002] In existing vision-based robot control or autonomous driving technologies, the agent needs to make decisions based on images captured by cameras. However, the captured image data often contains a large amount of redundant information irrelevant to the task (such as changes in lighting, dynamic backgrounds, and irrelevant textures). Although existing technologies utilize contrastive learning to extract features, they fail to effectively isolate the strong coupling relationships in image features, causing the control system to be unable to distinguish the motion features of the target object from the dynamic interference features of the background (i.e., the dimensionality mixing problem). This feature entanglement makes robots susceptible to being misled by background noise when facing complex and ever-changing unstructured environments, leading to technical problems such as slow convergence of control strategies, jitter in motion execution, and poor generalization ability in unfamiliar environments.

[0003] Thus, visual reinforcement learning was introduced into image processing. In existing visual reinforcement learning, the agent needs to extract a low-dimensional latent representation from high-dimensional image observations and output an action based on this representation to maximize the cumulative reward. The general process involves using a convolutional neural network as an encoder to encode the image into a feature vector, and then inputting this vector into a policy network and a value network.

[0004] Specifically, existing visual reinforcement learning primarily focuses on constructing discriminative representation spaces, mainly falling into two categories: 1. Auxiliary learning objectives: For example, CURL (Michael Laskin, Aravind Srinivas, and Pieter Abbeel. 2020. Curl: Contrastive unsupervised representations for reinforcement learning. In International conference on machine learning. PMLR, 5639–5650.) first introduced contrastive learning into visual reinforcement learning, generating discriminative representations by aligning embeddings of different reinforced views; ATC (Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. 2021. Decoupling representation learning from reinforcement learning. In International conference on machine learning. PMLR, 9870–9879.) injects sequence structure by maximizing the mutual information between current and future state representations; ADAT (Minbeom Kim, Kyeongha Rho, Yong-duk Kim, and Kyomin Jung. 2022. Action-driven contrastive representation for reinforcement learning. Plos one 17, 3 (2022)). e0265456.) proposes action-driven contrastive targets to better capture control-related features; SPR (Max Schwarzer, Ankesh Anand, RishabGoel, R Devon Hjelm, Aaron Courville, Aaron Courville and Philip Bachman. 2021. Data-Efficient Reinforcement Learning with Self-Predictive Representations. In International Conference on Learning Representations.)Explicitly predicting future latent features utilizes temporal consistency in the representation space; PSRL (Hyesong Choi, Hunsang Lee, Wonil Song, Sangryul Jeon, Kwanghoon Sohn, and Dongbo Min. 2023. Local-guidedglobal: Paired similarity representation for visual reinforcement learning. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 15072–15082.) proposes a locally guided global contrastive learning scheme that captures short-term dynamic continuity and long-range semantic similarity without relying on explicit dynamic prediction; TACO (Ruijie Zheng, Xiyao Wang, Yanchao Sun, Shuang Ma, Jieyu Zhao, Huazhe Xu, Hal Daumé III, and Furong Huang. 2023. TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning. Advances in Neural Information Processing Systems 36(2023),) explicitly predicts future latent features by utilizing temporal consistency in the representation space. (48203–48225.) uses temporally action-driven contrastive loss to align latent representations across time; CoIT (Sicong Liu, Xi Sheryl Zhang, Yushuo Li, Yifan Zhang, and Jian Cheng. 2023. On the data-efficiency with contrastive image transformation in reinforcement learning. In The Eleventh International Conference on Learning Representations.) focuses on invariance under image transformations to improve data efficiency. 2. Data Augmentation: For example, RAD (Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. 2020. Reinforcement learning with augmented data. Advances in neural information processing systems 33 (2020), 19884–19895.) first demonstrated that simple pixel-based augmentations (such as random cropping) can regularize training and stabilize Q-learning; DrQ (Denis Yarats, Ilya Kostrikov, and Rob Fergus. 2021. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International conference on learning representations.) further utilizes multi-view averaging of Q-objectives to improve data utilization; PlayVirtual (Tao Yu, Cuiling Lan, Wenjun Zeng, MingxiaoFeng, Zhizheng Zhang, and Zhibo Chen. 2021. Playvirtual: Augmenting cycle-consistent virtual trajectories for reinforcement learning). Advances in Neural Information Processing Systems 34 (2021), 5276–5289. effectively enriched the training sequence by using periodically consistent virtual scrolling for trajectory-level augmentation; subsequent work such as CycAug (Guozheng Ma, Linrui Zhang, Haoyu Wang, Lu Li, Zilin Wang, Zhen Wang, Li Shen, Xueqian Wang, and Dacheng Tao. 2023. Learning better with less: Effective augmentation for sample-efficient visual reinforcement learning.)Advances in Neural Information Processing Systems 36 (2023), 59832–59859.), SODA (Nicklas Hansen and Xiaolong Wang. 2021. Generalization in reinforcement learning by soft data augmentation. In 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 13611–13617.), SRM (Yangru Huang, Peixi Peng, Yifan Zhao, Guangyao Chen, and Yonghong Tian. 2022. Spectrum random masking for generalization in image-based reinforcement learning. Advances in Neural Information Processing Systems 35 (2022), 20393–20406.), CG2A (Siao Liu, Zhaoyu Chen, Yang Liu, Yuzheng Wang, Dingkang Yang, Zhile Zhao, Ziqing Zhou, Xie Yi, Wei Li, Wenqiang Zhang, et al. 2023. Improving generalization in visual reinforcement learning via conflict-aware gradient agreement augmentation. In Proceedings of the IEEE / CVF international conference on computer vision. 23436–23446.) and FGA3 (Jeong Woon Lee and Hyoseok Hwang. 2025. Fourier Guided Adaptive Adversarial Augmentation for Generalization in Visual Reinforcement Learning.(Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, 18110–18118.) explores domain-specific or frequency-domain augmentations to improve generalization ability. These schemes typically employ end-to-end training methods, attempting to implicitly process high-dimensional information.

[0005] Although existing technologies have improved sample efficiency to some extent, from the perspective of dimensional analysis, the following fundamental shortcomings still exist: 1. Severe dimensional redundancy in the feature space: While existing contrastive learning or data augmentation methods can learn discriminative features, they do not decouple these features at the dimensional level. In fact, even after training, there is still a high degree of information overlap (high correlation) between different dimensions in the feature space, resulting in a non-compact representation space. This not only wastes computational resources but also increases the training difficulty of downstream policy networks. 2. Inability to eliminate dimensional confounding factors: Not all information extracted from images is beneficial for decision-making. Existing methods often retain a large amount of task-irrelevant background or interfering information (i.e., confounding factors). Experiments show that randomly masking certain feature dimensions during testing can actually improve model performance, proving the objective existence of "harmful" information. Current technologies lack a mechanism to automatically identify and suppress these dimensions that are detrimental to decision-making; 3. Limited sample efficiency and generalization ability: Due to the existence of redundancy and interference mentioned above, the model needs more interactive data to filter out effective information, resulting in low sample efficiency in complex dynamic environments and poor generalization ability to environmental changes. Summary of the Invention

[0006] This invention provides an adaptive control method and system based on visual reinforcement learning, which improves the control accuracy and robustness of robots in dynamic disturbance environments.

[0007] To achieve the above objectives, the technical solution of the present invention includes the following:

[0008] An adaptive control method based on visual reinforcement learning, the method comprising: Construct a control model, which includes a target encoder, a target mapping layer, and a policy network; Construct training data; where each set of training data includes: the current time-step image, the current time-step action instruction, the current time-step reward, and the next time-step image; A control model is trained on the training data, and an online encoder, an online mapping layer, and a value network are combined to calculate the redundancy reduction loss and the reinforcement learning loss; wherein, the redundancy reduction loss is obtained by minimizing the correlation between different feature dimensions and maximizing the consistency of corresponding dimensions across views of the same image; The parameters of the control model, online encoder, online mapping layer and value network are updated based on redundancy reduction loss and reinforcement learning loss to obtain the trained control model. Control information of the target image is obtained based on the trained control model.

[0009] Furthermore, a control model is trained on the training data, and combined with an online encoder, an online mapping layer, and a value network, redundancy reduction loss and reinforcement learning loss are calculated, including: Different random data augmentation processes are applied to the image at the current time to obtain the first augmentation result and the second augmentation result. The first enhancement processing result and the second enhancement processing result are respectively input into the online encoder and the target encoder to obtain the online features and target features of the image at the current time; wherein, the target encoder parameters change with the online encoding parameters. The online features and target features are mapped based on the online mapping layer and the target mapping layer, respectively, to obtain the relationship between the online feature embedding representation and the target feature embedding. Based on the empirical cross-correlation matrix between the online feature embedding representation and the target feature embedding, the redundancy reduction loss is obtained. The online features are masked and combined with the action instruction at the current time step, the reward at the current time step, and the image at the next time step to obtain the reinforcement learning loss.

[0010] Furthermore, based on the empirical cross-correlation matrix between the online feature embedding representation and the target feature embedding, the redundancy reduction loss is obtained, including: Construct an empirical cross-correlation matrix between the online feature embedding representation and the target feature embedding; wherein, in this empirical cross-correlation matrix, the i-th... Line number Row matrix elements , This refers to the batch sample index. This refers to the embedding representation of online features. The Middle Sample In the Values ​​in each feature dimension This refers to the embedding representation of target features. The Middle Sample In the Values ​​in each feature dimension; Computational redundancy reduces losses , This indicates the first weight.

[0011] Furthermore, the online features are masked and combined with the current action instruction, the current reward, and the next image to obtain the reinforcement learning loss, including: A learnable dimensional mask is defined based on the feature space dimension of the online feature embedding representation; The embedded representation of the online feature is multiplied element-wise with the dimensional mask to obtain the weighted representation; The weighted representation is input into the policy network and the value network to obtain the reinforcement learning loss.

[0012] Furthermore, the update process of the dimensional mask includes: Virtual updates of parameters for the control model, online encoder, online mapping layer, and value network are performed based on reinforcement learning loss to obtain exploratory parameters; Based on the tentative parameters, the gradient of the reinforcement learning loss with respect to the dimensionality mask is calculated, and the dimensionality mask is updated based on this gradient.

[0013] Furthermore, control information of the target image is obtained based on the trained control model, including: Obtaining target image features based on a target encoder; The target image features are fed into the target mapping layer to obtain the target image embedding; The target image is embedded and fed into the policy network to obtain the control information of the target image.

[0014] An adaptive control system based on visual reinforcement learning, the system comprising: The model building module is used to build a control model, which includes a target encoder, a target mapping layer, and a policy network. The data construction module is used to construct training data; each set of training data includes: the current time-step image, the current time-step action instruction, the current time-step reward, and the next time-step image. The model training module is used to train the control model on the training data and, in conjunction with the online encoder, online mapping layer, and value network, calculate the redundancy reduction loss and reinforcement learning loss. The redundancy reduction loss is obtained by minimizing the correlation between different feature dimensions and maximizing the consistency of corresponding dimensions across views of the same image. Based on the redundancy reduction loss and reinforcement learning loss, the parameters of the control model, online encoder, online mapping layer, and value network are updated to obtain the trained control model. The information generation module is used to obtain control information of the target image based on the trained control model.

[0015] A computer device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the adaptive control method based on visual reinforcement learning as described above.

[0016] A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the adaptive control method based on visual reinforcement learning as described above.

[0017] A computer program product, characterized in that, when the computer program product is run on a computer device, it causes the computer device to execute the adaptive control method based on visual reinforcement learning described above.

[0018] Compared with the prior art, the present invention has at least the following beneficial effects.

[0019] 1. Feature Space Decoupling for More Efficient Information Representation: Through a dimensional redundancy suppression mechanism, this invention decouples high-dimensional visual signals into statistically independent feature components during the feature extraction stage. Compared to existing methods, the features generated by this invention encode non-overlapping information entropy in each dimension, resulting in more obvious differences between dimensions. This decoupling reduces the complexity of the feature space, making it easier for subsequent policy learning to find the optimal solution.

[0020] 2. Automatically suppressing interference information and improving decision robustness: Through a meta-learning-based dimensional masking mechanism, this invention can automatically identify and suppress feature channels containing background noise or interference information (i.e., confounding factors). After applying this invention, the robot only focuses on the key visual regions that contribute to the execution of actions when making decisions, thereby significantly improving the control accuracy and robustness of the robot in dynamic interference environments.

[0021] 3. Significantly improves sample efficiency and generalization performance: It can quickly learn effective control strategies with less interaction data, shortening the training cycle of the agent in the real physical environment.

[0022] 4. Wide applicability: The reinforcement learning part of this invention does not change the original RL algorithm architecture. Integrating it into different baseline algorithms can consistently improve the performance of the original algorithms, demonstrating high engineering practical value. Attached Figure Description

[0023] Figure 1 Flowchart of a visual reinforcement learning algorithm based on dimensionality analysis. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0025] This invention decouples feature dimensions through an explicit redundancy elimination mechanism and automatically identifies and utilizes key decision features by combining a meta-learning-based dimension masking mechanism, thereby significantly improving the sample efficiency and generalization ability of visual reinforcement learning. The invention is further illustrated below using a robotic arm visual grasping control task as an example. Figure 1 As shown, the present invention includes the following steps.

[0026] Step 1: Construct the control model.

[0027] The control model of this invention consists of a target encoder, a target mapping layer, and a policy network. The target encoder is responsible for encoding the target image, the target mapping layer maps the encoding result into an embedded representation, and the policy network outputs control information for the target image based on this embedded representation.

[0028] Step 2: Construct training data.

[0029] This embodiment uses a robotic arm's visual grasping control task as an example to illustrate the specific application of the present invention in visual reinforcement learning. In the robot control system (which can be a MuJoCo simulation environment or a real robotic arm operating platform), an RGB depth camera is configured as a visual sensor.

[0030] Status Acquisition: The camera acquires real-time images of the scene where the robotic arm and the target object are located at the current moment as the observation status. The image not only includes the robotic arm and the target object, but may also contain complex background textures, varying lighting, and moving distractions unrelated to the task.

[0031] Interactive execution: The robotic arm executes actions based on the generated commands. (e.g., joint torque or end-effector velocity) executes motion, and the environment provides feedback on the image of the next moment. and reward value (e.g., distance from the target, whether the capture was successful).

[0032] Experience storage: storing quadruples Stored in the experience replay buffer for subsequent network training.

[0033] Step 3: Train the control model on the training data, and combine the online encoder, online mapping layer and value network to calculate the redundancy reduction loss and reinforcement learning loss.

[0034] (1) Feature decoupling encoding based on dual-view enhancement.

[0035] Sample a batch of image data from the experience playback buffer, for the same image Perform two different random data augmentation operations (such as random pruning) to generate two views. and This step is to simulate the robot's observations under different viewpoints or lighting conditions, thereby enhancing the model's robustness.

[0036] Will and Input to the online encoder respectively and target encoder In this process, the image is mapped to a low-dimensional feature vector. and The target encoder and the online encoder have the same initial parameters. After each batch of data is trained, the online encoder... It will change significantly through gradient descent. At this point, the target encoder... It will slowly follow the changes in the online encoder, as shown in the formula: in It is a coefficient that is close to 1 or not less than a set threshold to ensure that the target encoder evolves smoothly and stably.

[0037] (2) Dimensional redundancy suppression.

[0038] To decouple features, a projection layer is introduced. Input online projection layer to obtain ,Will Input target projection layer to obtain .

[0039] calculate and Empirical cross-correlation matrix between , where matrix elements The calculation formula is as follows: in, This refers to the batch sample index. This refers to the embedded representation of the output of an online projector. The sample at the th Values ​​in each feature dimension This refers to the embedded representation output by the target projector. The sample at the th Values ​​in each feature dimension.

[0040] Constructing a redundancy reduction loss function : By minimizing This forces the encoder to learn an orthogonally decoupled feature space. For example, the first-dimensional feature only represents the position of the robotic arm, and the second-dimensional feature only represents the color of the target object, thus avoiding feature entanglement.

[0041] (3) Apply dimensional mask.

[0042] In a decoupled feature space, not all dimensions are useful for control decisions (e.g., the color feature of a background wall is distracting). This step automatically filters key features using a learnable dimensionality mask. A learnable parameter vector (dimensionality mask) is defined. , Representation of embedding The feature space dimension, For dimensional indexing.

[0043] The output of the online encoder With mask Perform element-wise multiplication to obtain the weighted representation. : In this process, feature dimensions corresponding to background noise and interference are assigned weights close to 0, thus being filtered out; while feature dimensions corresponding to the robotic arm's end effector and gripping point are assigned high weights, thus being amplified. The inputs are fed into the subsequent policy network and Q-network, which output specific robotic arm joint control commands and calculate the main loss for the reinforcement learning task. .

[0044] In one embodiment, an actor-critic framework is used to calculate the main loss. : in, It is the loss of the actor network (strategy network), which aims to maximize long-term returns; It is the loss of the commentator network (Q network or value network), which aims to accurately predict the value of a state or action; and These are the parameters for the actor network and the critic network, respectively. Indicates weight, Image representing the next moment The corresponding weighted representation.

[0045] Step 4: Update the parameters of the control model, online encoder, online mapping layer, and value network based on redundancy reduction loss and reinforcement learning loss to obtain the trained control model.

[0046] This invention employs an alternating update strategy: 1. Regular Updates: Fixed Mask Based on total loss Update the parameters of the online encoder, projection layer, and policy network. This allows the network to learn how to extract features and perform actions.

[0047] 2. Mask element update: First, the network parameters are virtually updated using the current reinforcement learning loss to obtain tentative parameters. (Not for practical application, only for calculating gradients): Then, based on the trial parameters ,calculate For the mask The gradient, and update : in, and These are the learning rates for the network parameters and the dimension mask, respectively. yes For current network parameters The first derivative, Based on test parameters Calculated reinforcement learning loss function For the mask The derivative of . This mechanism makes The direction of updates can reflect whether the model will perform better in future steps if certain dimensions are emphasized.

[0048] Step 5: Obtain control information of the target image based on the trained control model.

[0049] The trained policy network with a masking mechanism is deployed in the robotic arm controller. The controller receives camera images in real time, performs feature decoupling and masking filtering, and then directly outputs motor control signals to drive the robotic arm to complete the grasping task.

[0050] In summary, to address the feature coupling problem, this invention introduces a redundancy suppression regularization term. This scheme does not rely on negative samples; instead, it calculates the cross-correlation matrix of the two augmented views in the projection space, forcing this matrix to approximate the identity matrix. This not only guarantees the invariance of the views under augmentation (diagonal elements tend to 1) but also forces off-diagonal elements to tend to 0, thereby statistically achieving decorrelation of the feature dimensions and obtaining a compact feature representation.

[0051] To address the issue of dimensionality mixing, this invention designs a learnable mask vector with the same dimension as the feature vector. This mask is applied to the feature representation element-wise to weight the importance of different dimensions. This allows the model to dynamically amplify the signal of key features while suppressing features that are irrelevant to the task or harmful.

[0052] Since it is difficult to accurately assess the contribution of each dimension to long-term returns directly through a single backpropagation, this invention employs a two-layer optimization strategy. First, a "trial update" of the policy network is performed based on the current mask. Then, based on the performance of the updated model on the RL loss, the contribution of each dimension to the long-term returns is calculated. The second-order gradient. This mechanism gives the mask the ability to "learn to recognize key features".

[0053] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. An adaptive control method based on visual reinforcement learning, characterized in that, The method includes: Construct a control model, which includes a target encoder, a target mapping layer, and a policy network; Construct training data; where each set of training data includes: the current time-step image, the current time-step action instruction, the current time-step reward, and the next time-step image; A control model is trained on the training data, and an online encoder, an online mapping layer, and a value network are combined to calculate the redundancy reduction loss and the reinforcement learning loss; wherein, the redundancy reduction loss is obtained by minimizing the correlation between different feature dimensions and maximizing the consistency of corresponding dimensions across views of the same image; The parameters of the control model, online encoder, online mapping layer and value network are updated based on redundancy reduction loss and reinforcement learning loss to obtain the trained control model. Control information of the target image is obtained based on the trained control model.

2. The method according to claim 1, characterized in that, The control model is trained on the training data, and combined with an online encoder, an online mapping layer, and a value network, the redundancy reduction loss and reinforcement learning loss are calculated, including: Different random data augmentation processes are applied to the image at the current time to obtain the first augmentation result and the second augmentation result. The first enhancement processing result and the second enhancement processing result are respectively input into the online encoder and the target encoder to obtain the online features and target features of the image at the current time; wherein, the target encoder parameters change with the online encoding parameters. The online features and target features are mapped based on the online mapping layer and the target mapping layer, respectively, to obtain the relationship between the online feature embedding representation and the target feature embedding. Based on the empirical cross-correlation matrix between the online feature embedding representation and the target feature embedding, the redundancy reduction loss is obtained. The online features are masked and combined with the action instruction at the current time step, the reward at the current time step, and the image at the next time step to obtain the reinforcement learning loss.

3. The method according to claim 2, characterized in that, Based on the empirical cross-correlation matrix between the online feature embedding representation and the target feature embedding, the redundancy reduction loss is obtained, including: Construct an empirical cross-correlation matrix between the online feature embedding representation and the target feature embedding; wherein, in this empirical cross-correlation matrix, the i-th Line number Row matrix elements , This refers to the batch sample index. This refers to the embedding representation of online features. The Middle Sample In the Values ​​in each feature dimension This refers to the embedding representation of target features. The Middle Sample In the Values ​​in each feature dimension; Computational redundancy reduces losses , This indicates the first weight.

4. The method according to claim 2, characterized in that, The online features are masked and combined with the current action instruction, current reward, and next image to obtain the reinforcement learning loss, including: A learnable dimensional mask is defined based on the feature space dimension of the online feature embedding representation; The embedded representation of the online feature is multiplied element-wise with the dimensional mask to obtain the weighted representation; The weighted representation is input into the policy network and the value network to obtain the reinforcement learning loss.

5. The method according to claim 4, characterized in that, The update process of the dimensional mask includes: Virtual updates of parameters for the control model, online encoder, online mapping layer, and value network are performed based on reinforcement learning loss to obtain exploratory parameters; Based on the tentative parameters, the gradient of the reinforcement learning loss with respect to the dimensionality mask is calculated, and the dimensionality mask is updated based on this gradient.

6. The method according to claim 1, characterized in that, Control information of the target image is obtained based on the trained control model, including: Obtaining target image features based on a target encoder; The target image features are fed into the target mapping layer to obtain the target image embedding; The target image is embedded and fed into the policy network to obtain the control information of the target image.

7. An adaptive control system based on visual reinforcement learning, characterized in that, The system includes: The model building module is used to build a control model, which includes a target encoder, a target mapping layer, and a policy network. The data construction module is used to construct training data; each set of training data includes: the current time-step image, the current time-step action instruction, the current time-step reward, and the next time-step image. The model training module is used to train the control model on the training data and, in conjunction with the online encoder, online mapping layer, and value network, calculate the redundancy reduction loss and reinforcement learning loss. The redundancy reduction loss is obtained by minimizing the correlation between different feature dimensions and maximizing the consistency of corresponding dimensions across views of the same image. Based on the redundancy reduction loss and reinforcement learning loss, the parameters of the control model, online encoder, online mapping layer, and value network are updated to obtain the trained control model. The information generation module is used to obtain control information of the target image based on the trained control model.

8. A computer device, characterized in that, The computer device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the adaptive control method based on visual reinforcement learning as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the adaptive control method based on visual reinforcement learning as described in any one of claims 1-6.

10. A computer program product, characterized in that, When the computer program product is run on a computer device, the computer device performs the adaptive control method based on visual reinforcement learning as described in any one of claims 1-6.

Images