Lightweight visual decision interactive learning method and system based on knowledge migration

Abstract

The invention discloses a lightweight visual decision interactive learning method and system based on knowledge migration, and aims to realize efficient migration and reuse of pre-trained visual knowledge by constructing association constraints of a feature distance and a decision value, integrating domain and task knowledge and designing a lightweight interactive mechanism. The problems that in the prior art, pre-training features are directly reused, and a traditional feature distillation method is disjointed in feature and decision-making requirements, weak in cross-task generalization, low in dynamic environment efficiency and the like are solved. A pre-trained visual representation model is used as a knowledge source, a downstream feature coding module is used for learning feature representation of an adaptive decision task, a distance regularization constraint mechanism is designed to bind a feature distance with a decision value, and finally, a stable and high-generalization decision model is output through a training strategy fusing constraint loss and distillation loss. According to the method, an interactive learning mechanism is constructed through a low-dimensional interactive interface, so that the decision model can realize a lightweight decision training target based on an external dynamic instruction optimization strategy.

Description

TECHNICAL FIELD

[0001] The present application relates to a lightweight visual decision interaction learning system based on knowledge transfer, which is suitable for efficient training of agent decision strategy in a visual input scene, and aims to improve the learning efficiency, generalization and stability of the decision model through feature distance constraint; belongs to the cross technical field of computer vision and reinforcement learning. BACKGROUND

[0002] In the technical field of agent learning decision strategy based on visual input (such as image captured by camera) (i.e. visual reinforcement learning), improving the analysis efficiency of visual information of agent, accelerating the training process of decision strategy and enhancing its adaptability in various scenes are the core research direction of this field. At present, using pre-trained visual representation model (i.e. model pre-trained by large-scale image data, which can extract key features of image, hereinafter referred to as pre-trained model) to assist decision strategy learning is the mainstream technical means to solve the above problems, which has been widely used in practical scenes such as robot operation control, automatic driving, intelligent game, etc.

[0003] In the prior art, the use of pre-trained model is mainly realized by the following two ways: 1. Directly reuse the feature output of pre-trained model This way fixes the pre-trained model (such as residual network model for image classification, cross-modal contrast learning model for scene understanding, etc.) as a "feature extraction tool". Its working process is: the pre-trained model first processes the original visual input (such as the operation scene image captured by the robot camera), and outputs simplified visual features (numerical vectors reflecting object position and shape); then these features are input into the reinforcement learning decision network (such as deep value network for calculating action value, near-optimal strategy optimization network for optimizing strategy, etc.), to assist learning the "correspondence between scene and action". 2. Transfer the knowledge of pre-trained model through feature distillation This way makes the feature extraction network (hereinafter referred to as downstream network) dedicated to visual reinforcement learning imitate the feature output of pre-trained model through "distillation" mechanism. During training, the same visual input is sent to both of them, and the downstream network features are forced to be consistent with the pre-trained model features in value and distribution through loss function (such as mean square error, cosine similarity).

[0004] Although the prior art improves the efficiency of visual reinforcement learning, it has the following key problems, resulting in insufficient stability of decision-making strategy and weak generalization ability: lack of correlation between features and decision value: the features of the pre-training model only reflect visual similarity and are not associated with "decision value" (expected return of action). Insufficient adaptability across tasks and environments: there is an essential difference between the pre-training model and the downstream task goal: pre-training aims to "identify image content" (such as distinguishing between a cup and a plate), while visual reinforcement learning aims to "complete the task through a sequence of actions" (such as "put the cup in the plate"). This difference makes it difficult for pre-training features to adapt to different tasks. Low training efficiency in dynamic environments: in dynamic scenarios (such as autonomous driving with moving pedestrians, robot control after object collision), existing features have poor stability, resulting in the need for a large number of samples for the agent to learn a stable strategy.

[0005] The core of the above problems is that existing technologies only pursue "visual level feature similarity" and do not establish a correlation between features and decision value, task goal, and environmental dynamics, resulting in the inability to fully utilize the advantages of pre-training models. Therefore, it is necessary to bind features to decision requirements through explicit constraints to improve the stability and generalization of the strategy. SUMMARY

[0006] The purpose of the present application is to solve the problems of low learning efficiency from scratch and poor effect of traditional direct reuse of pre-training models and feature distillation methods in existing visual decision tasks. By extracting the constraint relationship between feature distance and decision value, a visual decision training method and system based on distance regularization feature distillation are built to improve the stability, generalization, and training efficiency of the agent's decision-making strategy.

[0007] Technical solution: A lightweight visual decision interaction learning system based on knowledge transfer aims to build an association constraint between feature distance and decision value to achieve efficient reuse of pre-training visual knowledge and solve the problems of "disconnection between features and decision requirements", "weak cross-task generalization", and "low efficiency in dynamic environments" in existing technologies that directly reuse pre-training features and traditional feature distillation methods. The core technical solution is to use a pre-training visual representation model as a knowledge source, learn feature representations that adapt to decision tasks through a downstream feature encoding module, design a distance regularization constraint mechanism to bind feature distance to decision value, and finally output a stable and highly generalizable decision model through a training strategy that combines constraint loss and distillation loss.

[0008] The system includes the following modules: (1) Pre-training visual representation module: a visual feature extraction model pre-trained on large-scale image data is adopted, the pre-trained visual feature extraction model parameters are fixed and do not participate in training, the input is the original visual input (such as a scene image), and the output is a fixed dimension pre-trained feature reflecting visual information (reflecting basic visual information such as object shape and spatial position). This module serves as a passive source of visual knowledge, avoiding the decision model from learning visual information from scratch and reducing training costs.

[0009] (2) Downstream feature encoding module, including feature encoder and action fusion unit: Feature encoder: a "convolutional neural network + fully connected layer" structure is adopted to process the original visual input, and output preliminary visual features, which refer to the key information extracted from the visual input; Action fusion unit: the preliminary visual features and the current action instruction are fused by vector splicing, and are sent to a linear transformation layer together, and a representation that can be directly input to the downstream module is encoded. The goal is to make the features contain both visual information and action influence signals on decision-making.

[0010] (3) Distance regularization constraint module, which accepts the features obtained by the pre-training visual representation module and the downstream feature encoding module as input, which are two constraint loss functions in form, loss function one and loss function two. The goal is to extract information from the pre-training network through a distillation loss function suitable for decision-making tasks, so that the decision-making features not only inherit the visual understanding ability of the pre-training model, but also adapt to the action logic of the decision-making task. Specifically, the goal of loss function one is to control the upper limit of the distance of the decision-making features, so that the feature output is consistent with the visual input and the action input, avoiding the confusion of similar visual features and different action outputs. The goal of loss function two is to bind the distance of the decision-making features with the decision-making value, so that the feature output is consistent with the decision-making value input, and the feature distance changes synchronously with the difference in decision-making value.

[0011] (4) Decision learning and distillation module, including distillation feature generation unit and decision network: distillation feature generation unit: adopts a fully connected network structure, inputs the pre-training features and action instructions, and outputs distillation features with the same dimension as the decision-making features, realizing the transfer of pre-training knowledge to the decision-making task; decision network: based on the decision-making features, the mapping relationship of "feature - action - reward" is learned, and the final decision-making strategy is output; loss optimization: fusion of condition one constraint loss, condition two constraint loss and distillation loss, the parameters of the downstream feature encoding module and the decision network are updated through back propagation (the pre-training model parameters are fixed).

[0012] It should be noted that the knowledge transfer and interactive learning functions of the system are realized relying on the distillation mechanism of the decision learning and distillation module, without the need for additional independent modules: on the one hand, knowledge fusion is naturally achieved through the distillation process. The distillation feature generation unit takes the pre-training features output by the pre-training visual representation model as the knowledge source. When the pre-training features and action instructions are fused to generate distillation features, the visual domain knowledge contained in the pre-training model is simultaneously transferred to the downstream feature encoding module and the decision network, so that the decision features naturally carry the pre-training knowledge, realizing the deep integration of knowledge and decision-making process; on the other hand, light interaction is realized relying on the parameter regulation characteristics of the distillation mechanism. After the external interaction instruction is encoded in low dimension, it only acts on the parameter fine-tuning of the distillation feature generation unit and the threshold adaptation of the distance regularization constraint module, without the need to call the pre-training model itself for complex calculation, which not only reduces the computational burden of the interaction process, but also guarantees the intervention response efficiency, meeting the light interaction demand.

[0013] The implementation logic of the system includes: (1) Selective reuse mechanism of pre-training knowledge: Unlike the traditional "direct reuse of pre-training features" method, the present application indirectly transfers pre-training knowledge through the distillation feature generation unit: while learning the visual information of the pre-training features, the downstream feature encoding module establishes the association between feature distance and decision value through the two constraint losses mentioned above, so that the decision features not only inherit the visual understanding ability of the pre-training model (such as recognizing object edges and spatial relationships), but also adapt to the action logic of specific tasks (such as robot grasping trajectory and vehicle turning rules), solving the problem of "poor adaptability caused by differences between pre-training and decision-making goals".

[0014] (2) Double distance regularization constraint: Condition one constraint: By setting an upper limit for the decision feature distance (pre-training feature distance + action distance), it is ensured that scenes with similar vision and similar actions output similar features, and scenes with similar vision but different actions output differentiated features. This is achieved through a loss function. Condition two constraint: The decision feature distance is bound to the "decision value correlation quantity" (current reward difference + discounted next state feature distance), so that the feature distance reflects the immediate reward difference of the action and its impact on the future state.

[0015] (3) Loss optimization strategy that combines constraint and distillation: The total loss consists of three parts, achieving the dual goals of "constraining feature distance + transferring pre-training knowledge": Total loss = Condition one constraint loss + Condition two constraint loss + Dynamic distillation loss, Dynamic distillation loss: composed of cosine similarity and smooth L1 loss (measuring feature value consistency) according to a preset weight, used to align the decision features and the distillation features; Dynamic switching mechanism: when T2 is greater than T1, the distillation loss is disabled to avoid invalid alignment when the decision-making requirement difference is too large; otherwise, the distillation loss is enabled to ensure effective transfer of pre-training knowledge.

[0016] The training process of the decision model: the training process is developed around "feature extraction - distance calculation - constraint optimization - parameter update", and the core is to make the downstream modules meet the constraints through iterative optimization. First, initialization: load the pre-trained model, initialize the downstream feature encoding module, decision network and hyperparameters. Then, feature extraction: the pre-trained model outputs pre-trained features, the downstream module outputs decision features, and the distillation unit outputs distillation features. Then, distance and value calculation: calculate the pre-trained feature distance, action distance, decision feature distance, and decision value difference, discounted next decision feature distance, etc. Then, loss calculation: calculate the constraint losses one and two and the distillation loss, and sum to get the total loss. Then, parameter update: fix the pre-trained visual representation model parameters, update the decision learning and distillation modules through backpropagation, and minimize the total loss. Repeat the "feature extraction - distance calculation - constraint optimization - parameter update" steps until the defined training deployment is reached, and output the trained decision model.

[0017] A lightweight visual decision interaction learning method based on knowledge transfer, comprising the following steps: Step 1, select a visual feature extraction model pre-trained on large-scale image data, fix the pre-trained visual feature extraction model parameters, and do not participate in training, input the original visual input into the pre-trained visual feature extraction model, and the output of the pre-trained visual feature extraction model is a fixed-dimensional pre-trained feature that embodies visual information.

[0018] Step 2, use the "convolutional neural network + fully connected layer" structure to process the original visual input and output preliminary visual features.

[0019] Step 3, fuse the preliminary visual features and the current action instruction through vector splicing, and input them into the linear transformation layer together to encode the representation that can be directly input into the downstream module. The goal is to make the features contain both visual information and action influence signals on decisions.

[0020] Step 4, take the outputs of steps 2 and 3 as inputs to construct two constraint functions, loss function one and loss function two. Loss function one controls the upper limit of the decision feature distance to avoid decision confusion caused by similar visual but different actions outputting similar features; loss function two binds the decision feature distance with the decision value to make the feature distance change synchronously with the decision value difference.

[0021] Step 5, using a full connection network structure, the input is the pre-training feature and the action instruction, and the output is the distillation feature consistent with the decision feature dimension, realizing the migration of pre-training knowledge to the decision task; learning the mapping relationship of "feature-action-reward" based on the decision feature, outputting the final decision strategy; fusing the first constraint loss, the second constraint loss and the distillation loss, updating the parameters of the decision learning and distillation module (the pre-training model parameters are fixed) through back propagation, and minimizing the total loss.

[0022] The decision learning and distillation module realizes the functions of knowledge migration and interactive learning through a distillation mechanism: The knowledge migration is achieved through a distillation process, and the distillation feature generation unit takes the output of the pre-training visual representation model as the knowledge source, converts the pre-training visual knowledge into distillation features suitable for the decision task, and inputs the downstream feature encoding module and the decision network, replacing the additional knowledge module to realize the migration of pre-training knowledge to the decision process. The interactive learning is realized by relying on the distillation mechanism, and the external instruction is received through a low-dimensional interactive interface, the instruction is encoded and applied to the parameter fine-tuning of the distillation feature generation unit and the threshold adjustment of the distance regularization constraint module, without calling the pre-training model itself, which reduces the computational overhead while ensuring the interactive response efficiency, and realizes the lightweight interactive intervention.

[0023] The method is realized in the same way as the system, and will not be repeated here.

[0024] Compared with the prior art, the present application has many advantages: no need to learn visual information from scratch, reduce training cost through pre-training knowledge reuse; two-layer constraint solves the "feature and decision disconnection" problem, improves the strategy stability; dynamic distillation mechanism adapts to different task logic, improves the cross-task generalization rate; the constraint feature distance fluctuates with the necessity of decision, and the utilization efficiency of dynamic environment samples is improved. In summary, the present application is suitable for various visual decision tasks such as robot control and automatic driving, and effectively solves the defects of the prior art.

[0025] Beneficial effects: the present application builds a lightweight visual decision interaction learning method and system based on knowledge transfer by establishing the correlation constraint of feature distance and decision value, effectively solves the problem that the features are disconnected with the decision requirements in the existing visual decision task, significantly improves the stability of the decision strategy, reduces the misjudgment caused by similar features but different decision requirements; at the same time, by adapting the pre-training knowledge and the downstream task dynamically, the generalization ability across tasks and environments is enhanced, the redundant cost of learning from scratch is avoided, and the performance loss during scene switching is reduced; in the dynamic environment, by constraining the correlation between feature fluctuation and decision necessity, the training efficiency is greatly improved, the sample consumption caused by invalid exploration is reduced, and the overall scheme can efficiently reuse the pre-training knowledge without complex adjustment, simplifying the training process of visual decision task and having wider application scope. In addition, the present application builds a lightweight interaction learning process, so that the decision model can respond to external interaction instructions in real time, avoiding the adaptation defects of traditional static training in dynamic scenes; and the interaction process only involves parameter fine-tuning, without increasing additional computational burden, and cooperates with the knowledge transfer mechanism to realize the comprehensive advantages of "efficient learning + dynamic interaction + lightweight", further expanding the application value in real-time interaction scenes. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is the flowchart of the training when the system is deployed in the embodiment of the present application; Figure 2 is the flowchart of the execution when the system is deployed in the embodiment of the present application; Figure 3 is the structure diagram of the specific network of the system in the embodiment of the present application; Figure 4 is a schematic diagram of an execution device for a visual decision interaction learning system in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The present application will be further illustrated below in conjunction with specific embodiments, which should be understood as merely illustrating the present application and not limiting the scope of the present application. After reading the present application, those skilled in the art can make various equivalent modifications to the present application, which fall within the scope defined by the claims attached hereto.

[0028] The specific embodiments of the present application are described in detail in conjunction with the accompanying drawings of Figure 1 (training flowchart), the accompanying drawings of Figure 2 (execution flowchart), the accompanying drawings of Figure 3 (network structure diagram) and the accompanying drawings of Figure 4 (execution device diagram). The present embodiment is applicable to various visual decision tasks, and the description of its technical details is sufficient to enable those skilled in the art to reproduce the technical solution of the present application without creative labor.

[0029] The embodiment realizes the adaptation of pre-trained visual knowledge and visual decision tasks by constructing a "distance regularization feature distillation" framework. The core is to establish the correlation between feature distance and decision value through two layers of distance constraints, which avoids learning from scratch while solving the defects of traditional reuse and distillation methods. Among them, distance regularization feature distillation uses distance regularization features to realize knowledge distillation, which is a way of knowledge transfer, which means using distance regularization features to train a neural network to make its output close to another existing neural network. The core idea of knowledge distillation and policy distillation in deep reinforcement learning developed from the twice learning method proposed by Professor Zhou Zhihua and Professor Jiang Yuan in 2004 in the KTDE journal.

[0030] The implementation details are described step by step according to system modules and training process as follows: 1 Implementation of pre-trained visual representation module 1.1 Pre-trained model selection and configuration A visual feature extraction model pre-trained on a large-scale image dataset is selected, and its parameters are fixed and not involved in training. This model receives raw visual input (such as a scene image), and outputs a fixed-dimensional basic feature vector (hereinafter referred to as "pre-trained feature") through convolution, pooling, and other operations. The vector contains basic information of the visual scene (such as object shape, spatial position relationship, etc.).

[0031] 1.2 Pre-trained feature extraction process After obtaining the raw visual input, first perform standardization processing on the input (such as normalizing the pixel value to a pre-set range), and then input it into the pre-trained model. The model outputs the pre-trained feature through layer-by-layer feature conversion and stores it in the memory for downstream module calling.

[0032] 2 Implementation of downstream feature encoding module 2.1 Feature encoder structure and function The feature encoder uses a combination of deep learning network structure convolutional neural network and fully connected layer to process the original visual input: first extract the local visual features in the input through the convolutional layer, and then convert them into fixed-dimensional preliminary visual features through the fully connected layer, ensuring that the features can reflect the key information of the visual input and adapt to subsequent fusion operations.

[0033] 2.2 Implementation of action fusion unit The action fusion unit receives the preliminary visual features output by the feature encoder and the action instructions in the current decision task (encoded and converted into vector form), and realizes fusion through feature splicing and linear transformation: splicing the preliminary visual features and action vectors into a joint feature vector, and then converting it into a decision feature with the same dimension as the pre-trained feature through a fully connected layer, so that the decision feature contains both visual information and the influence of action on decision.

[0034] 3 Distance regularization constraint module implementation 3.1 Distance metric method Euclidean distance is used as the unified metric method. The distance calculation formula for any two vectors is the sum of the square of the difference of each dimension of the vector. Specifically, it includes the following four types of distances: 3.1.1 Pre-training feature distance: calculate the Euclidean distance between the pre-training features corresponding to the two visual inputs, which is used to measure the similarity of the visual inputs; 3.1.2 Action distance: calculate the Euclidean distance between the two action instruction encoding vectors, which is used to measure the difference between the action instructions; 3.1.3 Decision feature distance: calculate the Euclidean distance between the two decision features, which is used to measure the difference between the decision features.

[0035] 3.1.4 Reward distance: calculate the Euclidean distance between the two reward results, which is used to measure the difference between the reward results; 3.2 Implementation of condition one constraint 3.2.1 Threshold calculation: calculate the threshold T1, which is the sum of the pre-training feature distance d(ψ(x),ψ(y)) and the action distance d(a,b), i.e. T1=d(ψ(x),ψ(y))+d(a,b); here x, y represent visual features, a, b represent decision actions, and d represents the distance function, represents the pre-training visual representation module; 3.2.2 Loss function construction: design a constraint loss Loss1, the formula is the square of the linear rectifier function ReLU result, where the input of the linear rectifier function is the "difference between the decision feature distance and T1"; when the decision feature distance is greater than T1, Loss1 produces a penalty, forcing the decision feature distance not to exceed T1, avoiding the feature being too similar when the visual is similar but the action is different, i.e. . represents the encoding network for visual features and decision actions.

[0036] 3.3 Implementation of condition two constraint 3.3.1 Decision value correlation quantity calculation: calculate the current reward difference , which is the absolute value of the difference between the immediate rewards corresponding to the two actions; calculate the next state pre-training feature distance, which is the pre-training feature distance of the next visual input generated after the execution of the two actions, and multiply the next state distance by the discount factor to reflect the influence weight of the future state on the current decision ; represents the decision value of executing decision action a under visual feature x. Specifically, it is the absolute value of the difference between the immediate rewards corresponding to the two actions; is the discount factor, and x' represents the visual feature of the next time of visual feature x.

[0037] 3.3.2 Threshold calculation: Calculate the threshold , whose value is the sum of the reward difference and the discounted next state distance; 3.3.3 Loss function construction: Design the constraint loss Loss2, whose formula is “the square of the decision feature distance and the difference value”, which forces the decision feature distance and the decision value difference to be consistent, so that the feature distance changes synchronously with the decision value difference, i.e. .

[0038] 4 Implementation of decision learning and distillation module 4.1 Implementation of distillation feature generation unit The distillation feature generation unit adopts a fully connected network structure, with the pre-training features and action instructions (converted into vectors through encoding) as inputs. Through feature splicing and linear transformation, it outputs distillation features consistent with the decision feature dimension, which are used to realize the transfer of pre-training knowledge to decision tasks.

[0039] 4.2 Implementation of distillation loss 4.2.1 Loss composition: The distillation loss is composed of cosine similarity (measuring feature direction consistency) and smooth L1 loss (measuring feature value consistency) combined according to the preset weight, where the weight coefficient can be set to 0.1 (to balance the influence of the two); that is, the distillation loss , β is the fusion weight, and g represents the encoding network for the pre-training visual feature and the decision action after encoding.

[0040] 4.2.2 Dynamic switching mechanism: Set the on-off coefficient, which is 0 (distillation loss does not take effect) when is greater than ; otherwise, the coefficient is 1 (distillation loss takes effect), avoiding invalid feature alignment when the decision requirement difference is too large.

[0041] 4.3 Implementation of total loss and parameter update 4.3.1 Total loss calculation: The total loss is the sum of loss function one, loss function two, and the distillation loss multiplied by the on-off coefficient; 4.3.2 Parameter update process: Use an optimizer (such as Adam optimizer) to update the parameters of the downstream feature encoding module, distillation feature generation unit, and decision network (the pre-training model parameters are fixed); process a preset number of samples in each training round, calculate the gradient of the total loss with respect to each parameter through backpropagation, and update the parameters in the gradient descent direction until the training converges (such as the loss fluctuation of consecutive preset rounds being less than a threshold value).

[0042] 5. Implementation of decision strategy generation After training, for new visual input, the decision-making process is as follows: 5.1 Obtain new visual input and generate a set of candidate actions; 5.2 For each candidate action, output the corresponding decision feature through the downstream feature encoding module; 5.3 The decision network calculates the expected return (reflecting the likelihood of successful execution of the action) of each action based on the decision feature; 5.4 Select the action with the highest expected return as the final decision and execute it.

[0043] 6. Summary of implementation effects Through the above implementation, the present application achieves the following effects in visual decision-making tasks: pre-trained knowledge is efficiently reused, avoiding the redundant cost of learning from scratch; the association constraint between feature distance and decision value solves the problem of disconnection between features and decision requirements in traditional methods, improving the stability of the decision strategy; the dynamic distillation mechanism enhances the adaptation ability across tasks and environments; at the same time, the training efficiency in dynamic environments is significantly improved.

[0044] A person skilled in the art can adjust the pre-trained model type, network structure parameters and hyperparameters according to specific task requirements, and all can achieve the technical effects of the present application.

Claims

1. A knowledge transfer based lightweight visual decision interaction learning system, characterized in that, The method comprises the following modules: A pre-training visual representation module: a visual feature extraction model pre-trained by image data is adopted, and the parameters of the visual feature extraction model are fixed; the input of the visual feature extraction model is an original visual input, and the output is a fixed-dimension pre-training feature embodying visual information; A downstream feature encoding module comprising a feature encoder and an action fusion unit: The feature encoder: a "convolutional neural network + fully connected layer" structure is adopted to process the original visual input, and an initial visual feature is outputted; The action fusion unit: the initial visual feature and the current action instruction are fused by vector splicing, and the spliced vector is inputted into a linear transformation layer to obtain a representation which can be directly inputted into a downstream module through encoding; A distance regularization constraint module: the features obtained by the pre-training visual representation module and the downstream feature encoding module are inputted into the distance regularization constraint module, which is in the form of two constraint loss functions, i.e., loss function one and loss function two, and the target is to extract information in the pre-training network through a distillation loss function suitable for a decision task; the target of the loss function one is to make the feature output consistent with the visual input and the action input, so as to avoid the confusion of the decision action caused by the similar visual and different actions; the target of the loss function two is to make the feature output consistent with the decision value input, so that the feature distance changes synchronously with the value obtained by the decision; A decision learning and distillation module comprising a distillation feature generation unit and a decision network: The distillation feature generation unit: a fully connected network structure is adopted, the input is the pre-training feature and the action instruction, the output is a distillation feature with the same dimension as the decision feature, and the migration of the pre-training knowledge to the decision task is realized; The decision network: based on the decision feature, the mapping relationship of "feature-action-reward" is learned, and the final decision strategy is outputted; Loss optimization: the distillation loss function in the distance regularization constraint module and the loss function of the decision process itself are fused, and the parameters of the downstream feature encoding module and the decision network are updated through back propagation. 2.The knowledge transfer based light visual decision interaction learning system according to claim 1, wherein, The feature encoder extracts the local visual features in the original visual input through the convolutional neural network, and then converts the local visual features into the initial visual features with fixed dimensions through the fully connected layer; The action fusion unit splices the initial visual features and the action vector into a joint feature vector, and then converts the joint feature vector into the decision feature with the same dimension as the pre-training feature through the fully connected layer, so that the decision feature contains both the visual information and the influence of the action on the decision. 3.The knowledge transfer based light visual decision interaction learning system according to claim 1, wherein, In the distance regularization constraint module, the Euclidean distance is adopted as the measurement of the following four distances: Pre-training feature distance: the Euclidean distance between the pre-training features corresponding to two visual inputs is calculated, which is used to measure the similarity of the visual inputs; Action distance: the Euclidean distance between the two action instruction encoding vectors is calculated, which is used to measure the difference between the action instructions; Decision feature distance: the Euclidean distance between two decision features is calculated, which is used to measure the difference between the decision features; Reward distance: the Euclidean distance between the values obtained after the two decisions is calculated, which is used to measure the difference between the decision values. 4.The knowledge transfer based light visual decision interaction learning system according to claim 1, wherein, The implementation of the condition one constraint comprises: Threshold calculation: Calculate the threshold T1, which is the pre-trained feature distance d ( (x), The sum of (y) and the action distance d(a,b), i.e., T1=d( (x), (y))+d(a,b), where x and y represent visual features, a and b represent decision actions, and d represents the distance function. This represents a pre-trained visual representation module; Loss function construction: design condition one constraint, that is, constraint loss function one, the formula is the square of the result of the linear rectifier function ReLU, wherein the input of the linear rectifier function is "the difference between the decision feature distance and T1"; when the decision feature distance is greater than T1, Loss1 produces a penalty, forcing the decision feature distance to be less than or equal to T1, that is ; here indicates the encoding network for visual features and decision actions.

5. The knowledge transfer based light visual decision interaction learning system according to claim 1, wherein, The implementation of the condition two constraint comprises: Decision value correlation quantity calculation: calculate the current reward difference Here represents the decision value of performing the decision action a under the visual feature x, which is the absolute value of the difference between the instant rewards corresponding to the two actions; calculate the next state pre-training feature distance, that is, the pre-training feature distance of the next visual input generated after the execution of the two actions, and multiply the next state distance by the discount factor to reflect the influence weight of the future state on the current decision Here is the discount factor, and x' represents the visual feature of the next time of the visual feature x; Threshold calculation: Calculate threshold whose value is the sum of the reward difference and the discounted next state distance; Loss function construction: design condition two constraints, that is, constrain the loss function two, the formula is "the square of the distance between the decision characteristics and the difference of T2". Through this loss, the decision characteristic distance is forced to be consistent with T2, so that the characteristic distance changes synchronously with the decision value difference, that is .

6. The knowledge transfer based light visual decision interaction learning system according to claim 1, wherein, The distillation loss is combined by a cosine similarity cos_sim and a smooth L1 loss L1 according to a preset weight; that is, the loss function three is also called a distillation loss Here, β is a fusion weight, and g represents an encoding network for the pre-trained visual feature and the decision action after encoding.

7. The knowledge transfer based light visual decision interaction learning system according to claim 1, wherein, In the implementation of the total loss and the parameter update, the total loss is calculated The total loss is calculated as the sum of the loss function one, the loss function two and the loss function three multiplied by the opening coefficient; the opening coefficient is set to 0 when T2 is greater than T1, otherwise the opening coefficient is 1; The parameter update process: using an optimizer to update the parameters of the downstream feature encoding module, the distillation feature generation unit and the decision network; a preset number of samples are processed in each round of training, the gradient of the total loss with respect to each parameter is calculated through back propagation, and the parameters are updated in the gradient descent direction until the training converges.

8. The knowledge transfer based light visual decision interaction learning system according to claim 1, wherein, In the implementation of the decision strategy generation, for a new visual input, the decision process is as follows: (1) obtaining the new visual input, generating a candidate action set; (2) for each candidate action, output the corresponding decision feature through the downstream feature encoding module; (3) the decision network calculates the expected return of each action based on the decision feature; (4) selecting the action with the highest expected return as the final decision and executing it.

9. A lightweight visual decision interaction learning method based on knowledge transfer, characterized in that, The method comprises the following steps: Step 1, selecting a visual feature extraction model pre-trained by large-scale image data, fixing the parameters of the pre-trained visual feature extraction model and not participating in training, taking the original visual input as the input of the pre-trained visual feature extraction model, and outputting the pre-trained visual feature extraction model as a fixed-dimensional visual information pre-trained feature; Step 2, using the "convolutional neural network + fully connected layer" structure to process the original visual input and output preliminary visual features; Step 3, fusing the preliminary visual features and the current action instruction through vector splicing and inputting them into a linear transformation layer to encode the representation that can be directly input into the downstream module, the goal being to make the features contain both visual information and action influence signals on decision-making; Step 4, taking the outputs of steps 2 and 3 as inputs to construct two constraint functions, namely loss function one and loss function two, loss function one controls the upper limit of the distance between decision features to avoid decision confusion caused by similar visual features but different actions; loss function two binds the distance between decision features with the decision value, so that the feature distance changes synchronously with the difference in decision value; Step 5, using a fully connected network structure, inputting the pre-trained features and action instructions, and outputting distillation features with the same dimension as the decision features to realize the transfer of pre-trained knowledge to the decision task; learning the mapping relationship between "features - actions - rewards" based on the decision features to output the final decision strategy; fusing the constraint losses of conditions one and two and the distillation loss to update the parameters of the downstream feature encoding module and the decision network through back propagation to minimize the total loss.

10. The knowledge transfer based light visual decision interaction learning system according to claim 1, wherein, The decision learning and distillation module synchronously realizes the functions of knowledge transfer and interactive learning through the distillation mechanism: The knowledge transfer is achieved through the distillation process, and the distillation feature generation unit takes the output of the pre-trained visual representation model as the knowledge source to convert the pre-trained visual knowledge into distillation features suitable for the decision task, which are input into the downstream feature encoding module and the decision network to replace the additional knowledge module and realize the transfer of pre-trained knowledge to the decision process; The interactive learning is realized by relying on a distillation mechanism, receives external instructions through a low-dimensional interactive interface, and adjusts the threshold value of the parameter fine-tuning and distance regularization constraint module of the distillation feature generation unit after the instructions are encoded, without calling the pre-trained model itself.

Images