A visual representation migration method and apparatus

Abstract

The application discloses a visual representation migration method and device. The method comprises the following steps: obtaining a pre-training model trained by using a short-term strategy optimization algorithm, the pre-training model being applied to a first environment, and the pre-training model being capable of identifying a first visual representation feature of the first environment; learning a prompt encoder in a second environment based on the first visual representation feature, the prompt encoder being used for converting a second visual representation feature of the second environment into the first visual representation feature; and converting the second visual representation feature of the second environment into the first visual representation feature capable of being identified by the pre-training model based on the prompt encoder. The above scheme enables the pre-training model to quickly learn the prompt encoder in a new environment, thereby realizing efficient visual representation migration and strategy migration, enabling the pre-training model to be transferred to a new environment which needs to solve the same basic problem but has different visual inputs, and reducing the cost of re-training the model in the new environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the automatic driving technology, more particularly, to a visual representation migration method and device. BACKGROUND

[0002] In recent years, with the wide application of automatic driving, there are many methods for better deploying models in real environments.

[0003] DRL((Deep Reinforcement Learning), deep reinforcement learning) method has been applied to games, robots, automatic driving and other fields and has achieved good results. In most fields, the DRL method carefully designs the reward function and network structure for a specific task, and obtains the model by continuously interacting with the environment to collect a large amount of data and iterative training. It is difficult for the DRL algorithm to transfer these pre-trained models to new environments, which need to solve the same problem but have different visual inputs. For example: the performance of the pre-trained model will be greatly reduced or even completely collapsed in these new environments, and common environmental changes include irregular shapes or color changes.

[0004] The change of visual representation in the new environment causes the pre-trained model to be unable to adapt to the new environment, thereby causing the model to be unable to be deployed in the real environment. The common method is to retrain or fine-tune the model from scratch, which has a high training cost, and also cannot restore the performance of the original pre-trained model. In addition, the reward function and network structure designed for the source environment may not be suitable for the new environment, so the strategy generated therefrom may still perform poorly in the new environment.

[0005] In order to solve this problem, some methods extract general domain space features in different environments as inputs for training models, which requires artificially designing complex loss functions and network structures so that the model can focus on cross-domain features. Although these methods can handle environmental changes, these methods all need to artificially redesign the loss function and retrain to achieve qualified performance in the new target environment. SUMMARY

[0006] Therefore, the present application provides the following technical solutions:

[0007] A visual representation migration method, comprising:

[0008] obtaining a pre-trained model trained using a near-term policy optimization PPO algorithm, the pre-trained model being applied to a first environment, and the pre-trained model being capable of recognizing a first visual representation feature of the first environment;

[0009] learning of the hint encoder in the second environment based on the first visual representation feature, the hint encoder being used to convert a second visual representation feature of the second environment into the first visual representation feature;

[0010] the second visual representation feature of the second environment is converted into the first visual representation feature recognizable by the pre-trained model based on the hint encoder.

[0011] Optionally, the learning of the hint encoder in the second environment based on the first visual representation feature comprises:

[0012] obtaining an initial feature conversion function;

[0013] initializing the weight of the initial feature conversion function through imitation learning to obtain the hint encoder with feature conversion function;

[0014] optimizing the hint encoder based on the first visual representation feature by collecting data of the second environment.

[0015] Optionally, the initializing the weight of the initial hint encoder through imitation learning to obtain the hint encoder with feature conversion function comprises:

[0016] fitting the initial feature conversion function through imitation learning of prior knowledge of a small data set in the second environment to obtain the hint encoder with feature conversion function.

[0017] Optionally, the optimizing the hint encoder by collecting data of the second environment comprises:

[0018] the hint encoder is constantly optimized by collecting observation data of the second environment and trying to output actions to a higher score until the score of the output action is the highest.

[0019] Optionally, wherein during the learning process of the hint encoder, the network structure and weight of the feature network, the policy network and the value network in the network model are the same as the corresponding network structure and corresponding weight in the pre-trained model.

[0020] A visual representation migration device comprises:

[0021] a model obtaining module configured to obtain a pre-trained model trained using a near-episode policy optimization PPO algorithm, the pre-trained model being applied to a first environment, and the pre-trained model being capable of recognizing a first visual representation feature of the first environment;

[0022] an encoder learning module configured to learn a prompt encoder for converting second visual representation features of a second environment into the first visual representation features based on the first visual representation features;

[0023] a visual migration module configured to convert the second visual representation features of the second environment into the first visual representation features recognizable by the pre-trained model based on the prompt encoder.

[0024] Optionally, the encoder learning module comprises:

[0025] a function obtaining module configured to obtain an initial feature conversion function;

[0026] a weight initialization module configured to initialize weights of the initial feature conversion function by imitation learning to obtain the prompt encoder with the feature conversion function;

[0027] an encoder optimization module configured to optimize the prompt encoder based on the first visual representation features by collecting data of the second environment.

[0028] Optionally, the weight initialization module is specifically configured to guide fitting of the initial feature conversion function by imitation learning of prior knowledge of a small data set in the second environment to obtain the prompt encoder with the feature conversion function.

[0029] Optionally, the encoder optimization module is specifically configured to control the prompt encoder to be constantly optimized by collecting observation data of the second environment and trying to output actions to a higher score until a score of the output actions is the highest.

[0030] Optionally, the prompt encoder is constantly optimized by collecting observation data of the second environment and trying to output actions to a higher score until a score of the output actions is the highest.

[0031] Via the technical solution, the embodiment of the application discloses a visual representation migration method and device, the method comprises: obtaining a pre-training model trained by using a near-segment strategy optimization PPO algorithm, the pre-training model is applied to a first environment, and the pre-training model can identify first visual representation features of the first environment; learning of a prompt encoder in a second environment based on the first visual representation features, the prompt encoder is used for converting second visual representation features of the second environment into the first visual representation features; and converting the second visual representation features of the second environment into the first visual representation features identifiable by the pre-training model based on the prompt encoder. The above scheme enables the pre-training model to quickly learn the prompt encoder in a new environment, thereby realizing efficient visual representation migration and policy migration, enabling the pre-training model to be transferred to a new environment that needs to solve the same basic problem but has different visual inputs, and reducing the cost of retraining the model in the new environment. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0033] Figure 1 A flowchart of a visual representation migration method disclosed by the embodiment of the present application;

[0034] Figure 2 A flowchart of learning of a prompt encoder disclosed by the embodiment of the present application;

[0035] Figure 3 An implementation architecture schematic diagram of a visual representation migration scheme disclosed by the embodiment of the present application;

[0036] Figure 4 A structural schematic diagram of a visual representation migration device disclosed by the embodiment of the present application. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0038] Figure 1 A flowchart of a visual representation migration method disclosed by the embodiment of the present application. Referring toFigure 1 As shown, the visual representation migration representation method can include:

[0039] Step 101: obtaining a pre-trained model trained using a near-episode policy optimization PPO algorithm, the pre-trained model being applied to a first environment, the pre-trained model being capable of identifying a first visual representation feature of the first environment.

[0040] The pre-trained model refers to a model that is pre-trained in a first environment and can obtain high performance in the first environment. In the embodiment of the application, the pre-trained model can be a deep reinforcement learning DRL model.

[0041] Step 102: learning a prompt encoder in a second environment based on the first visual representation feature, the prompt encoder being used to convert a second visual representation feature of the second environment into the first visual representation feature.

[0042] In the application, the visual representation of the pre-trained model needs to be transferred from the first environment to the second environment, so that the pre-trained model only needs to be trained once in the source environment. On the other hand, the prompt method is introduced through the natural language processing task in the implementation, so that the downstream task can reuse the inference ability of the pre-trained model with a lower cost. The prompt method tries to reuse the pre-trained model by adding a prompt label for the downstream task. For example, the first environment is a car game environment, and the second environment is an automatic driving environment.

[0043] In one example, the pre-trained model in natural language processing: the model is trained in a large natural language processing data set to obtain a model with the highest natural language understanding performance, and the model is used as a pre-trained model. Assuming that the pre-trained model can perform a cloze test on a certain English text. For example: I----this movie. The input pre-trained model can obtain the output I love this movie or I hate this movie, etc.; now facing a new task, it is necessary to identify the sentiment type of the text, but the cost of retraining is too high, so it is desired to reuse the pre-trained model in 2. The prompt method can convert the new task into the task type supported by the pre-trained model in 2 by adding a label.

[0044] For example: judging the sentiment of x='I love this movie', expecting the output y=positive;

[0045] Construct a template (prompt template / label) [x]. Overall it was a [z] movie.

[0046] The above a is a hint template or called a hint mark, and the input x is filled in as [x] in a in this step to construct a new text type input: I love this movie. Overall, it was a [Z] movie.

[0047] At this time, the text data type obtained in b belongs to the cloze type, so the text data in b can be input into the pre-trained model in 2 to obtain the correct output result, such as: good.

[0048] After the above steps, an output good can be obtained, and the final output result y=positive is obtained after remapping. Here, good is mapped to positive, and bad is mapped to negative.

[0049] Through the above process, it can be found that the new task can reuse the pre-trained model by adding a hint mark.

[0050] The scheme can apply hints to transfer visual representations from a target task to a source task, that is, from a second environment to a first environment. The hint encoder, which is the core of the algorithm, migrates features extracted from the second environment to the first environment, so that when facing various new environments, the D pre-trained model can be reused instead of being retrained or fine-tuned in the new environment.

[0051] All that is needed in the implementation of the scheme is to train the hint encoder, which learns the feature conversion rule from the target environment to the source environment. Specifically, a multi-layer convolutional network can be used to construct the hint encoder to fit the hint function.

[0052] Step 103: Convert the second visual representation feature of the second environment into the first visual representation feature recognizable by the pre-trained model based on the hint encoder.

[0053] After obtaining the hint encoder, the conversion of the visual input of the second environment to the visual input of the first environment can be completed, so that for each different new environment, the pre-trained model can be applied by constructing a corresponding hint encoder. The training process of the hint encoder consumes very low cost compared to other migration algorithms, and the training parameter amount is also very low. Moreover, the pre-trained model of the scheme maintains stable performance in different environments and always maintains high performance.

[0054] The visual representation migration method described in the embodiment can quickly learn the prompt encoder in a new environment, thereby realizing efficient representation migration and policy migration. The visual transfer prompt method is applied in DRL, and the pre-trained model is transferred to a new environment that needs to solve the same basic problem but has different visual inputs. The method specifies a prompt encoder in the pre-trained model to transmit the visual representation, which can reuse the pre-trained model and reduce the training cost. The PB-PPO algorithm adjusts the prompt encoder for the target task in a two-stage training pipeline. This method only needs to learn the prompt encoder of the new environment to quickly adapt to the new environment, further improving the migration efficiency.

[0055] Figure 2 The flowchart for learning the prompt encoder is disclosed in the embodiments of the present application. As shown in Figure 2 The learning of the prompt encoder in the second environment based on the first visual representation feature can include:

[0056] Step 201: Obtain an initial feature conversion function.

[0057] Step 202: Initialize the weight of the initial feature conversion function through imitation learning to obtain a prompt encoder with feature conversion function.

[0058] Step 203: Based on the first visual representation feature, optimize the prompt encoder by collecting data of the second environment.

[0059] In the implementation, the training of the encoder is divided into two stages. In the first stage (corresponding to step 202), a small data set in the new environment is used to initialize the weight of the prompt encoder through imitation learning, so that it has a primary feature conversion capability. In the second stage (corresponding to step 203), the DRL collects observation data of the new environment and tries to make the output action get a higher score, and then continuously optimizes the prompt encoder. In the two stages, the pre-trained model of the source environment is frozen, and only the weight of the prompt encoder is learned.

[0060] Specifically, the initialization of the weight of the initial prompt encoder through imitation learning to obtain a prompt encoder with feature conversion function can include: fitting the initial feature conversion function by using the prior knowledge of the small data set in the second environment through imitation learning to obtain a prompt encoder with feature conversion function.

[0061] Specifically, the optimization of the prompt encoder by collecting data of the second environment can include: the prompt encoder is continuously optimized by collecting observation data of the second environment and trying to output actions to a higher score until the score of the output action is the highest.

[0062] The method of the application scheme corresponds to the "prompt" method in the DRL training process, and a prompt encoder module of the DRL strategy is designed to serve as a prompt function in the "prompt" method. When facing a new environment, the prompt encoder converts the representation features to adapt to the pre-trained model in the source environment, thereby realizing the rapid migration of the strategy while maintaining the performance of the strategy. The strategy refers to the task behavior decision information contained in the pre-trained model. Compared with the traditional PPO method, the application scheme only needs to adjust the weight of the prompt encoder, instead of retraining the entire DRL model or manually designing a complex loss function for the new environment. The method can be extended to cross-domain and cross-modal tasks of high-performance DRL methods driven by small data, thereby reducing the training cost of the DRL algorithm and improving the deployment and migration efficiency.

[0063] Specifically, the reuse of the pre-trained model can be realized through the following process.

[0064] 1. Train a pre-trained strategy model for the task in environment A;

[0065] 2. When facing a new environment B, construct a prompt encoder to convert the input data of the B environment into a form acceptable to the A pre-trained model;

[0066] 3. Jointly fine-tune the data obtained in step 2 with the pre-trained model of A to optimize the training, so as to obtain the best prompt encoder parameters that can better convert B data, so that the A pre-trained model can accept the B data of the new environment and achieve the same performance;

[0067] 4. The output result obtained in step 3 needs to be remapped to meet the requirements of the B task. If the requirements between the environments are consistent, this process can be omitted. Experiments show that there is no obvious difference between the two.

[0068] The inventors of the application scheme have experimented with the algorithm in the CarRacing simulation environment, and the results show that PB-PPO has high representation migration performance.

[0069] Specifically, the simulation environment can be specified as a Markov Decision Process (MDP) problem, which can be described by the tuple <S, A, T, R, γ>. S and A represent the state and action sets, T represents a function mapping from the current state and action to the next state. R represents a reward function mapping, and γ represents a discount factor. Proximal Policy Optimization (PPO) is a model-free policy gradient algorithm for DRL. It is usually composed of two networks. One samples an effective action from A, i.e., a policy network, and the other takes a value for the state in S, i.e., a value function network.

[0070] Specifically, the PPO parameterized strategy πθ (a|s) and value function V θ (s), which share the same architecture except for the last layer. The training objective of the policy network in PPO is to minimize the policy loss:

[0071]

[0072] where is the empirical expectation value, is the expected advantage. r(θ) is the ratio of the current policy to the previous policy, and ∈ is a hyperparameter. The clip function is defined as: clip(μ, α, β) = max(min(μ, β), α).

[0073] The training objective of the PPO value network is to minimize the mean squared error:

[0074]

[0075] Wherein, the value network and the policy network are networks in the PPO algorithm, and are used to realize the policy optimization process in the PPO algorithm. The policy network is used to output the behavior to be taken in the current state, and the value network is used to evaluate the behavior output by the policy network in the current state. The two continuously output policies and evaluate the policies, so as to realize the optimization process of the whole PPO algorithm, so as to realize that the policy network can output the optimal behavior.

[0076] Figure 3 The implementation architecture schematic diagram of the visual representation migration scheme disclosed in the embodiment of the application is shown in Figure 3 As shown in the figure, the scheme learns the prompt encoder between the target environment and the source environment, and uses the feature conversion capability of the prompt encoder, so that the pre-trained model combined with the prompt encoder module can quickly obtain high performance in the new environment. It consists of three parts:

[0077] (1) DRL pre-training in the source environment: a pre-trained model with high performance is trained in the original environment using the PPO algorithm.

[0078] (2) PB-PPO prompt encoder initialization: the weights of the prompt encoder are initialized by imitation learning. The weights of the prompt encoder are initialized in the new environment under the implementation of the task driven by the expert data, so that it has basic feature conversion capability.

[0079] (3) PB-PPO prompt encoder optimization: in this stage, DRL continuously optimizes the prompt encoder by collecting data in the new environment, and tries to make the output action get higher reward, and finally obtains the prompt encoder for migration.

[0080] Wherein, the DRL pre-training in the source environment:

[0081] Policy learning and training are performed in the source environment based on the PPO algorithm. The policy trained in the source environment is used as the pre-trained model π θ (α|s), where a is an action and s is observation data in the source environment. s' represents observation data in the target environment. The weights of the pre-trained model are directly reused in stage 1 and stage 2 without retraining or fine-tuning. As shown in Figure 3 a.

[0082] Prompt encoder for feature conversion in new environment:

[0083] Changes in the DRL environment can cause the trained model to be unable to be deployed. The present scheme proposes a prompt encoder for feature conversion. The objective of the encoder is to fit a feature conversion function h θ (s') using a convolutional neural network (CNN), as shown in Figure 3 d. The learned optimal conversion function is denoted as In the two stages, the training process of the prompt encoder is driven by different environments, aiming to learn an approximation of the conversion function closest to the source environment.

[0084] Stage 1: Prompt encoder initialization through imitation learning

[0085] During random exploration, DRL has a large amount of invalid exploration. Inspired by the text task, human prior knowledge is introduced to guide the learning of the prompt encoder. Therefore, in order to reduce the search space, imitation learning is used to initialize the parameters of the prompt encoder, that is, through imitation learning in the new environment, the prompt encoder learning process is guided to fit the initial feature conversion function h(s') relying on the prior knowledge in a small amount of expert data. Experiments show that effective initialization can speed up the learning speed of the prompt encoder in stage 2 and the entire task.

[0086] Specifically, Figure 3 b shows stage 1, which includes a prompt encoder network, a feature network, a policy network, and a fully connected layer (FC). Among them, the network structures of the feature network and the policy network remain unchanged when facing various new environments, and their weights come from the pre-trained model in the source environment, as shown by the light dashed line in Figure 3 . Their weights are frozen during the training process of stage 1, as shown by the dashed box in Figure 3 b. The action output of the DRL network is continuous, but when imitation learning is used in stage 1, the actions corresponding to the small amount of expert data input are discrete, so the FC layer is needed to convert the output policy network into a discrete form. When training stage 1 in the new environment, the network only needs a small amount of expert data and very short training time to obtain the initialization learning weights of the prompt encoder.

[0087] Phase 2: Continuously optimize the hint encoder by the proximal policy optimization algorithm

[0088] In phase 1, an effective hint encoder initialization h(s') is obtained. However, even with the DRL policy, the network still cannot obtain scores close to those in the source environment, which indicates that h(s') is not yet sufficient to transform the representation. Therefore, it is necessary to continue to optimize the hint encoder.

[0089] Figure 3 c shows the pipeline of phase 2, which includes: the hint encoder network, the feature network, the policy network and the value network. As in the first phase, the structures of the feature network, the policy network and the value network are the same as those used in the initial environment, and their network weights come from the pre-trained model of the source environment, as indicated by the light dashed line in Figure 3 . It is worth noting that the weights of the hint encoder obtained from training are loaded from the training results of the first phase, as indicated by the dark dashed line in Figure 3 . In addition, the feature network, the policy network and the value network in phase 2 also need to be frozen during training, as indicated by the dashed box in Figure 3 c. The new feature representation obtained after the hint encoder is as follows:

[0090] s * = h(s') (3)

[0091] Under the joint action of formula (1) and formula (2), when the reward value reaches the maximum value, the parameters learned by the hint encoder are the optimal results of the conversion function, denoted as h * (s').

[0092] The scheme first trains a pre-trained policy model with high performance in the original environment using the PPO algorithm, which is used for subsequent policy and representation migration. Then, a fast multi-stage hint encoder learning is performed in the environment where policy migration and representation migration are required. The learning cost of this process is much lower than that of directly training in the new environment. After training, the obtained hint encoder can realize the representation migration from the original environment to the target environment, and then realize the efficient migration of the policy without losing the performance of the policy. The saved hint encoder of the current environment can be used as a plug-and-play module in the actual deployment process to complete the deployment and migration work.

[0093] The prompt encoder is used for prompting and converting the visual representation, so that the converted representation can be used for the pre-trained model of the original environment, and parameters of the prompt encoder are obtained in a multi-stage training process in the new environment. In the multi-stage training process, the original pre-trained model parameters are kept unchanged, and optimization of the prompt encoder module parameters is implemented by using the multi-stage. The prompt encoder module is obtained after the final training is completed. When deployed, the prompt encoder module is inserted into the head of the pre-trained model in the original environment, so that high performance in the new environment can be achieved.

[0094] For each of the above method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the order of the described actions, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0095] The above embodiments of the present application disclose the method in detail, and the method of the present application can be implemented by various forms of devices, so the present application also discloses a device, and specific embodiments are given below to be described in detail.

[0096] Figure 4 A structural schematic diagram of a visual representation migration device disclosed by an embodiment of the present application is shown in FIG. 4. Figure 4 As shown in FIG. 4, the visual representation migration device 40 can include:

[0097] The model obtaining module 401 is configured to obtain a pre-trained model trained by using a proximal policy optimization (PPO) algorithm, the pre-trained model being applied to a first environment, and the pre-trained model being capable of recognizing a first visual representation feature of the first environment.

[0098] The encoder learning module 402 is configured to learn a prompt encoder in a second environment based on the first visual representation feature, the prompt encoder being used for converting a second visual representation feature of the second environment into the first visual representation feature.

[0099] The visual migration module 403 is configured to convert the second visual representation feature of the second environment into the first visual representation feature recognizable by the pre-trained model based on the prompt encoder.

[0100] The visual representation migration device described in the embodiment can quickly learn the prompt encoder in a new environment, thereby realizing efficient representation migration and policy migration. The visual transfer prompt method is applied in DRL, and a pre-trained model is transferred to a new environment that needs to solve the same basic problem but has different visual inputs. The method specifies a prompt encoder in the pre-trained model to transmit visual representation, which can reuse the pre-trained model and reduce the training cost. The PB-PPO algorithm adjusts the prompt encoder for the target task in a two-stage training pipeline. This method only needs to learn the prompt encoder of the new environment to quickly adapt to the new environment, further improving the migration efficiency.

[0101] In one implementation, the encoder learning module includes: a function obtaining module configured to obtain an initial feature conversion function; a weight initialization module configured to initialize weights of the initial feature conversion function through imitation learning to obtain a prompt encoder having a feature conversion function; and an encoder optimization module configured to optimize the prompt encoder based on the first visual representation feature by collecting data of the second environment.

[0102] In one implementation, the weight initialization module is specifically configured to guide fitting of the initial feature conversion function through imitation learning of prior knowledge of a small data set in the second environment to obtain the prompt encoder having the feature conversion function.

[0103] In one implementation, the encoder optimization module is specifically configured to control the prompt encoder to be continuously optimized by collecting observation data of the second environment and attempting to output actions to a higher score until the score of the output actions is the highest.

[0104] In one implementation, the prompt encoder is continuously optimized by collecting observation data of the second environment and attempting to output actions to a higher score until the score of the output actions is the highest.

[0105] The specific implementation of each module in the visual representation migration device described above can refer to the content introduction of the corresponding part in the method embodiment, which will not be repeated here.

[0106] The visual representation migration device described in the above embodiment includes a processor and a memory. The model obtaining module, the encoder learning module, the visual migration module, the function obtaining module, the weight initialization module, the encoder optimization module, and the like in the above embodiment are stored in the memory as program modules, and the processor executes the above program modules stored in the memory to realize the corresponding functions.

[0107] The processor includes a core, and the core retrieves the corresponding program modules from the memory. The core can be set to one or more, and the processing of the revisit data is realized by adjusting the core parameters.

[0108] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0109] In an exemplary embodiment, a computer-readable storage medium is also provided, which can be directly loaded into the internal memory of a computer, and contains software code that, after being loaded and executed by the computer, can implement the steps shown in any embodiment of the visual representation transfer method described above.

[0110] In an exemplary embodiment, a computer program product is also provided, which can be directly loaded into the internal memory of a computer and contains software code. After being loaded and executed by the computer, the computer program can implement the steps shown in any embodiment of the visual representation transfer method described above.

[0111] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0112] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0113] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0114] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the appended claims are intended to cover all such modifications that do not depart from the true spirit and scope of the application. Therefore, the application is not limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A visual representation migration method, characterized by, The method comprises the following steps: obtaining a pre-trained model trained by using a short-segment policy optimization PPO algorithm, the pre-trained model being applied to a first environment, and the pre-trained model being capable of identifying first visual representation features of the first environment; learning a prompt encoder in a second environment based on the first visual representation features, the prompt encoder being used to convert second visual representation features of the second environment into the first visual representation features; converting the second visual representation features of the second environment into the first visual representation features capable of being identified by the pre-trained model based on the prompt encoder; wherein the learning of the prompt encoder in the second environment based on the first visual representation features comprises: obtaining an initial feature conversion function; initializing weights of the initial feature conversion function by imitation learning to obtain the prompt encoder with the feature conversion function; optimizing the prompt encoder by collecting data of the second environment based on the first visual representation features; wherein the initializing of the weights of the initial feature conversion function by the imitation learning to obtain the prompt encoder with the feature conversion function comprises: guiding the fitting of the initial feature conversion function by prior knowledge of a small data set in the second environment by the imitation learning to obtain the prompt encoder with the feature conversion function; wherein the optimizing of the prompt encoder by collecting the data of the second environment comprises: the prompt encoder is constantly optimized by collecting observation data of the second environment and trying to output actions to a highest score until the score of the output actions is the highest; wherein the weights of the pre-trained model are frozen during the learning process of the prompt encoder.

2. The visual representation migration method of claim 1, wherein, wherein, during the learning process of the prompt encoder, the network structures and the weights of the feature network, the policy network and the value network in the network model are the same as the corresponding network structures and the corresponding weights in the pre-trained model.

3. A visual representation migration apparatus characterized by, The method comprises the following steps: a model obtaining module is configured to obtain a pre-trained model trained by using a short-segment policy optimization PPO algorithm, the pre-trained model being applied to a first environment, and the pre-trained model being capable of identifying first visual representation features of the first environment; an encoder learning module is configured to learn a prompt encoder in a second environment based on the first visual representation features, the prompt encoder being used to convert second visual representation features of the second environment into the first visual representation features; a visual migration module is configured to convert the second visual representation features of the second environment into the first visual representation features capable of being identified by the pre-trained model based on the prompt encoder; a function obtaining module is configured to obtain an initial feature conversion function; a weight initializing module is configured to initialize weights of the initial feature conversion function by imitation learning to obtain the prompt encoder with the feature conversion function; an encoder optimizing module is configured to optimize the prompt encoder by collecting data of the second environment based on the first visual representation features. The weight initialization module is specifically configured to guide fitting of the initial feature conversion function by imitating prior knowledge of a small data set in the second environment to obtain a prompt encoder with a feature conversion function. The encoder optimization module is specifically configured to control the prompt encoder to constantly optimize by collecting observation data of the second environment and trying to output actions to a higher score until the score of the output actions is the highest. During the learning process of the prompt encoder, the device freezes the weights of the pre-trained model.

4. The visual representation migration apparatus of claim 3, wherein, The prompt encoder constantly optimizes by collecting observation data of the second environment and trying to output actions to a higher score until the score of the output actions is the highest.

Images