Method and electronic device for policy update

Abstract

A method and an electronic device for policy updating are provided. The method for policy updating includes determining a movement path of an agent based on input data according to a learned policy, obtaining feedback determined based on whether a target point is reached in response to the agent completing the movement, and updating a parameter of the learned policy based on the feedback.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2025-0009013, filed on January 21, 2025, with the Korean Intellectual Property Office, and Korean Patent Application No. 10-2025-0037380, filed on March 24, 2025, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to methods and apparatus for policy updates. Background Technology

[0003] Data-driven decision-making technologies can be applied across various fields. Specifically, technologies that enable systems to interact with their environment and make appropriate decisions can be used in areas such as automation, optimization, and autonomous systems. In these technologies, methods for determining predetermined actions based on input data can be employed. The decision-making process may involve using pre-trained models or methods that adjust actions by reflecting information from the environment. Furthermore, to improve the accuracy of decision-making, methods may evaluate results against a given objective and incorporate these evaluations.

[0004] Furthermore, obtaining optimal results using only typical decision-making methods can be challenging when the environment changes or new variables emerge. Additionally, while typical systems can correct or adjust decisions based on given information, they may not be able to do so by utilizing additional evaluation results. Therefore, such typical systems that make decisions based on input data may exhibit insufficient performance. Summary of the Invention

[0005] The present invention is provided in a simplified form to introduce the choice of concepts further described in the following detailed embodiments. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0006] In one or more general aspects, a processor-implemented method includes: determining a movement path of an agent based on input data according to a learned policy; receiving feedback based on whether a target point has been reached in response to the agent completing the movement; and updating parameters of the learned policy based on the feedback.

[0007] The steps to obtain the feedback may include: obtaining the feedback for each event. The step of updating the parameters may include: updating the gradient of the learned policy based on the feedback; and updating the parameters of the learned policy based on the updated gradient.

[0008] The step of updating the parameters may further include: randomly reversing a portion of the updated gradient.

[0009] The steps of reversing a portion of the updated gradient may further include: determining whether to reverse the updated gradient based on a predefined probability value; and reversing the sign of the updated gradient in response to determining that the updated gradient has been reversed.

[0010] The step of reversing the sign of the updated gradient may also include adjusting the magnitude of the reversed gradient according to predefined coefficients.

[0011] The step of updating the parameters may include updating parameters that include the weights of an artificial neural network model configured to perform a learning strategy.

[0012] Artificial neural network models can be implemented as visual language navigation (VLN) models based on reinforcement learning.

[0013] The steps for obtaining the feedback may include: obtaining positive feedback in response to the agent reaching the target point; and obtaining negative feedback in response to the agent not reaching the target point.

[0014] The steps of updating the parameters of the learning strategy may include: applying a first predefined weight to the parameters of the learning strategy in response to receiving positive feedback, and applying a second predefined weight to the parameters of the learning strategy in response to receiving negative feedback.

[0015] The steps to obtain the feedback may include obtaining continuous value feedback based on the distance between the agent's final position and the target point.

[0016] The method may include: determining whether to generate the feedback based on whether the agent has completed an event, wherein the event may include a series of steps, and each of the steps includes a corresponding action performed by the agent along a movement path, and wherein the step of obtaining the feedback may include: generating the feedback based on a comparison between the last step and the target point in response to determining that the feedback has been generated.

[0017] In one or more general aspects, an electronic device includes: one or more processors, including a processing circuit system; and a memory, including one or more storage media storing instructions, which, when executed individually or jointly by the one or more processors, cause the electronic device to: determine a movement path of an agent based on input data according to a learned policy; receive feedback based on whether a target point has been reached in response to the agent completing the movement; and update parameters of the learned policy based on the feedback.

[0018] In order to obtain the feedback, the execution of the instruction enables the electronic device to obtain the feedback for each event.

[0019] In order to update the parameters, the execution of the instruction enables the electronic device to: update the gradient of the learned policy based on the feedback; and update the parameters of the learned policy based on the updated gradient.

[0020] In order to update the parameters, the execution of the instruction may cause the electronic device to randomly reverse a portion of the updated gradient.

[0021] In order to reverse a portion of the updated gradient, the execution of the instruction enables the electronic device to: determine whether to reverse the updated gradient based on a predefined probability value; and in response to determining whether to reverse the updated gradient, reverse the sign of the updated gradient.

[0022] In order to reverse the sign of the updated gradient, the execution of the instruction causes the electronic device to update parameters including the weights of an artificial neural network model configured to perform a learning strategy.

[0023] In order to obtain the feedback, the execution of the instruction can cause the electronic device to: receive positive feedback in response to the agent reaching the target point; and receive negative feedback in response to the agent not reaching the target point.

[0024] In one or more general aspects, a processor-implemented method includes: determining whether to generate feedback based on whether an agent has completed an event, wherein the event may include a series of steps, and each of the steps may include a corresponding action performed by the agent; and generating feedback based on a comparison between the last step in the steps and a target point in response to determining that feedback should be generated.

[0025] Other features and aspects will become clear from the following detailed description, drawings, and claims. Attached Figure Description

[0026] Figure 1A An example of a visual language navigation (VLN) model according to one or more embodiments is shown.

[0027] Figure 1B Examples of events (episodes) in a VLN model according to one or more embodiments are shown.

[0028] Figure 2 Examples of policy update methods according to one or more embodiments are shown.

[0029] Figure 3 Examples of policy update methods according to one or more embodiments are shown.

[0030] Figure 4 An example of providing binary feedback in a policy update method according to one or more embodiments is shown.

[0031] Figure 5 Examples of detailed instructions and corresponding events executed in a policy update method according to one or more embodiments are shown.

[0032] Figure 6 An example is shown of the application of the stochastic gradient inversion (SGR) technique in a policy update method according to one or more embodiments.

[0033] Figure 7 Examples of policy update methods according to one or more embodiments are shown.

[0034] Figure 8 Examples of electronic devices according to one or more embodiments are shown.

[0035] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same or identical elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative dimensions, scale, and depiction of elements in the drawings may be exaggerated. Detailed Implementation

[0036] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations within and / or the sequence of operations described herein are merely examples and are not limited to the order set forth herein, but may be changed as will become clear upon understanding this disclosure, except for the order of operations within and / or the sequence of operations that must occur in a specific order. As another example, the order of operations and / or the sequence of operations within and / or the sequence of operations may be performed in parallel, except for at least a portion of the order of operations within and / or the sequence of operations that must occur in a specific order (e.g., a specific order). Furthermore, for greater clarity and conciseness, descriptions of features known upon understanding this disclosure may be omitted.

[0037] Although terms such as “first,” “second,” and “third,” or A, B, (a), (b), etc., may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. For example, each of these terms is not used to define the nature, order, or sequence of the corresponding component, assembly, region, layer, or part, but only to distinguish the corresponding component, assembly, region, layer, or part from other components, assemblies, regions, layers, or parts. Therefore, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0038] Throughout this specification, when a component or element is described as being "on," "connected to," "bonded to," or "joined to" another component, element, or layer, it may be directly (e.g., in contact with) "on" "in contact with" the other component, element, or layer, "connected to," "bonded to," or "joined to" the other component, element, or layer, or one or more other components, elements, or layers may reasonably be present in between. When a component, element, or layer is described as being "directly on," "directly connected to," "directly bonded to," or "directly joined to" another component, element, or layer, no other components, elements, or layers may be present in between. Similarly, expressions such as "between" and "immediately between," and "adjacent to" and "closely adjacent to" may also be interpreted as described above.

[0039] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. As a non-limiting example, the terms “comprising,” “including,” and “having” indicate the presence of the stated features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof, or optionally, alternative features, quantities, operations, components, elements, and / or combinations thereof. Furthermore, while one embodiment may illustrate the use of the terms “comprising,” “including,” and “having” to indicate the presence of the stated features, quantities, operations, components, elements, and / or combinations thereof, other embodiments may exist in which one or more of the stated features, quantities, operations, components, elements, and / or combinations thereof are not present.

[0040] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains, particularly in the context of understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having the same meaning as they have in the context of the relevant field, particularly in the context of this disclosure, and shall not be interpreted in an idealized or overly formal sense.

[0041] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more. Phrases such as “at least one of A, B, and C”, “at least one of A, B, or C” are intended to have a disjunctive meaning, and unless the corresponding description and embodiments require such a list (e.g., “at least one of A, B, and C”) to be interpreted as having a conjunctive meaning, these phrases “at least one of A, B, and C”, “at least one of A, B, or C” (e.g., each phrase may include any one of the individual corresponding items, all items listed together, and all possible combinations thereof) also include examples of one or more of each of A, B, and / or C (e.g., any combination of one or more of each of A, B, and C).

[0042] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein have been provided only to illustrate some of the many feasible ways in which the methods, apparatus, and / or systems described herein will be clear upon understanding the disclosure of this application. The term “may” (e.g., regarding what an example or embodiment may include or implement) is used herein with respect to examples or embodiments to indicate the existence of at least one example or embodiment that includes or implements such a feature, while not all examples are limited thereto. The terms “example,” “embodiment,” and “example embodiment” are used herein with the same meaning (e.g., the phrase “in an example or in one example” has the same meaning as “in an embodiment or in one embodiment” and “in an example embodiment or in an example embodiment”), and “one or more examples” has the same meaning as “one or more embodiments” and “one or more example embodiments.” Furthermore, each of the multiple or all individually described “example,” “embodiment,” “example embodiment,” and “multiple examples,” “multiple embodiments,” “multiple example embodiments” may be combined into the same embodiment in any combination.

[0043] The examples can be implemented as various types of products (such as, for example, personal computers, laptops, tablets, smartphones, televisions, smart home appliances, smart vehicles, self-service terminals, and wearable devices). The examples are described in detail below with reference to the accompanying drawings. In the drawings, the same reference numerals are used for the same elements.

[0044] Figure 1A An example of a visual language navigation (VLN) model according to one or more embodiments is shown.

[0045] Reference Figure 1A In the VLN model, agent 110, oracle (also known as "perfect decision maker" or "oracle") 120 and environment 130 can interact with each other, and agent 110 can determine actions based on observations in environment 130 and perform natural language communication with oracle 120.

[0046] Agent 110 can determine a movement path and move to a target point based on input data provided in environment 130. Agent 110 can be implemented as an autonomous driving robot, an artificial intelligence-based navigation system, a navigation system within a virtual simulation, etc. However, examples are not limited to these. Furthermore, agent 110 can move within environment 130 by utilizing a policy learned based on reinforcement learning, and can determine actions to achieve a predetermined goal. Agent 110 can also be referred to as an explorer or an autonomous system. The policy can be a function that determines which action the agent chooses in a predetermined state in a given environment.

[0047] Environment 130 may be the space in which agent 110 and oracle 120 interact with each other, and may include virtual environments, physical environments, and / or combinations thereof. Environment 130 may be an indoor space (e.g., a home or office) and / or an outdoor space (e.g., a road or park), and may also be implemented as a simulated virtual environment (e.g., a three-dimensional (3D) simulated space). Environment 130 may also be referred to as an environment map or workspace.

[0048] Observations can be information collected from the environment 130 by agent 110 and oracle 120, and may include images, distance sensor data, light detection and ranging (LiDAR) data, text information, etc. Agent 110 may acquire observations through cameras, sensors, and / or other input devices, and determine the direction of movement based on the acquired observations. Observations may also be referred to as input data or environmental data.

[0049] Actions can be operations performed on the environment 130 by agent 110 and oracle 120. Agent 110 can perform actions (such as moving (e.g., moving left, moving forward, turning in a predetermined direction) or stopping). Oracle 120 can also perform predetermined actions (such as highlighting predetermined points within the environment 130 and / or providing feedback on the target location). Actions can also be referred to as movement and / or control commands.

[0050] Oracle 120 can be a system that evaluates the navigation performance of agent 110 and provides feedback during the testing phase. Oracle 120 can operate based on feedback provided directly by a human, and / or an artificial intelligence-based system can perform the evaluation automatically. For example, oracle 120 can determine whether a target point has been reached to generate success or failure feedback, and can also provide distance-based evaluation results. Oracle 120 can also be referred to as an evaluation system or a feedback provider.

[0051] Natural language communication is a method of transmitting information between agent 110 and oracle 120, and may include sentences expressed in natural language. For example, oracle 120 may send a sentence (such as "move left") to agent 110. Natural language communication may include voice commands, text commands, and / or other language expressions, and may be combined with a speech recognition system. Natural language communication may also be referred to as language-based interaction or instruction delivery.

[0052] Agent 110, oracle 120, and environment 130 can interact with each other, and agent 110 and oracle 120 can collect observations from environment 130 and perform actions. In addition, agent 110 and oracle 120 can exchange information through natural language communication and update policies based on that information.

[0053] For example, environment 130 could be a 3D space generated by simulating the interior of a home. Agent 110 could be an autonomous mobile robot capable of performing tasks such as moving to a predetermined location within the home. Oracle 120 could be an artificial intelligence (AI) based evaluation system capable of analyzing the exploratory performance of agent 110 and providing feedback.

[0054] The agent 110 in its initial state can receive natural language instructions. For example, agent 110 can receive instructions (such as "move from the living room to the kitchen") from oracle 120. Instructions can be provided in the form of speech and / or text. Natural language communication can be expressed as commands and / or instructions.

[0055] The agent 110 can obtain observations from the environment 130. For example, observations can be information such as "currently in the living room", "there is a sofa in front", and "the kitchen is on the left". Observations can be provided in the form of red, green, and blue (RGB) images, depth information, location data, etc., and can also be expressed as sensor data or scene understanding.

[0056] The agent 110 can determine a movement path and execute actions based on a learned policy. For example, the agent 110 can move in ways such as "move forward 2 meters (m)" and "turn 90 degrees to the right and move 1m". The movement technique can be continuous navigation or grid-based movement, and the actions can be represented as movement commands or directional control.

[0057] As information in the environment 130 changes, the agent 110 can continuously update its observations. For example, the agent 110 can acquire new observation information (such as "currently at the kitchen entrance" or "there is a table in front") and redetermine its path based on the new observation information.

[0058] When agent 110 reaches or fails to reach the target point, oracle 120 can perform an evaluation. For example, evaluation results (such as "Successfully reached the target", "Did not reach the target, final position: corridor") can be generated. Oracle 120 can generate feedback by comparing agent 110's final position (or last step) with the target point, and the feedback can be provided as binary feedback (success = +1, failure = -1) or continuous distance-based rewards.

[0059] In policy update methods, the policy can be updated episode-by-episode, rather than modified at each step. For example, the policy update method can determine whether to generate feedback based on whether the agent has completed an event. For instance, agent 110 can adjust its policy based on feedback provided by oracle 120 in response to completing an entire movement path, without immediately modifying the policy during a single movement. Therefore, the methods and apparatus of one or more embodiments can prevent policies from becoming unstable due to short-term action changes and can improve policies more consistently. See also Figure 1B Examples of events described in detail.

[0060] Due to limitations in the training data, VLN models may fail to guarantee expected performance in real-world environments due to low generalization performance. While typical methods and apparatuses may apply test-time adaptive (TTA) to address this issue, this approach cannot effectively overcome domain disparities because it leads to error accumulation and overfitting problems. Therefore, methods and apparatuses of one or more embodiments may include more reliable policy update methods for performing VLN tasks.

[0061] One or more embodiments of the policy update method can address the domain discrepancy problem that occurs when testing VLN models using event-by-event feedback techniques. One or more embodiments of the policy update method can collect feedback from the test environment and adjust the policy based on the feedback, enabling the agent to operate stably in the new environment.

[0062] However, policy update methods are not necessarily intended to solve only the problems described above. Policy update methods can have a variety of other effects. For example, reinforcement learning-based policy networks in one or more embodiments can improve the adaptability and reliability of VLN models by dynamically updating policies using feedback. Furthermore, policy update methods are not limited to VLN models. For example, policy update methods can also be applied to robots that include autonomous agents operating in various environments, or other AI systems that perform decision-making based on learning-based policies and multimodal computer vision applications.

[0063] Figure 1B Examples of events in a VLN model according to one or more embodiments are shown. (Refer to...) Figure 1A The provided description is essentially the same. Figure 1B .

[0064] Reference Figure 1B An event can be defined as a series of navigation processes by which an agent 110 reaches a target point within an environment 130, and a step can be a unit in which the agent 110 performs a single action within an event. An event may include a series of steps, and each step may include a corresponding action performed by the agent along a movement path.

[0065] Events may include a predetermined number of steps that may vary depending on the complexity of the environment 130 and / or the navigation path. Figure 1B Three events (e.g., first event 140, second event 150, and third event 160) are shown as examples. The three events may each include four, three, and five steps, respectively. First event 140 may include a total of four steps, second event 150 may include a total of three steps, and third event 160 may include a total of five steps. Events may be referred to as navigation processes and / or navigation sequences, and steps may be referred to as navigation units and / or action frames.

[0066] For example, environment 130 may be assumed to be inside a home, and agent 110 may be asked or determined to move from a predetermined room to the kitchen. A first event 140 may include four steps (e.g., "living room → hallway → kitchen entrance → kitchen"), and a second event 150 may include three steps (e.g., "living room → kitchen entrance → kitchen"). Furthermore, a third event 160 may include five steps (e.g., "living room → hallway → study → kitchen entrance → kitchen"). Each event may include the same starting step (e.g., "living room") and the same ending step (e.g., "kitchen").

[0067] The number of steps in each event can vary depending on the structure of the environment 130, the navigation path, the location of obstacles, etc. In a predetermined event, there may be an optimal path that requires only a few steps to reach the target point. In another event, the number of steps may increase as obstacles are bypassed or the path is redefined. For example, even with the same starting point and the same target point, the number of steps performed by the agent 110 may vary depending on the strategy for learning the navigation path.

[0068] According to the example, policy updates can be performed event-by-event rather than in each individual step. For example, agent 110 can receive feedback from oracle 120 in response to reaching or not reaching a target point, and update the policy based on the feedback. By performing policy updates event-by-event rather than in each individual step, the methods and apparatus of one or more embodiments can achieve more stable learning than methods and apparatuses that change the policy for each individual action.

[0069] Figure 2 Examples of policy update methods according to one or more embodiments are shown. (Refer to...) Figure 1A and Figure 1B The provided description is also applicable Figure 2 .

[0070] Reference Figure 2 Agent 210 can perform actions based on input data, and oracle 220 can evaluate the actions of agent 210 and provide feedback to agent 210. The provided feedback can be used to update the policy of agent 210.

[0071] The policy update method described in the examples illustrates how a trained policy network (e.g., a VLN model) can be adapted during the testing phase (inference). For example, unlike a typical training process, one or more embodiments of the policy update method can operate such that the policy network can be adapted to a new environment by updating its policy based on feedback from a given environment and testing time. For instance, agent 210 can perform navigation based on the trained policy network and can receive feedback from oracle 220 in response to navigation completion. In this process, agent 210 can adjust its policy based on direct experience in the testing environment without requiring pre-trained data.

[0072] The policy update process, as illustrated in the example, may include using only feedback in a test environment without requiring a new dataset to adapt the model. Agent 210 may perform navigation in a given environment based on a pre-learned policy. In response to navigation completion, oracle 220 may evaluate the navigation result of agent 210 and provide feedback. The feedback may reflect whether agent 210 has reached the target point. For example, when the target point is reached, the feedback may be set to F=+1, and when the target point is not reached, the feedback may be set to F=-1.

[0073] Agent 210 can directly update its policy based on feedback provided by oracle 220. Policy updates can be performed by applying stochastic gradient reversion (SGR), and can be performed by reversing or scaling a portion of the gradient according to predetermined probabilities. In this way, the methods and apparatus of one or more embodiments can avoid overfitting to predetermined failure modes and can adjust the policy to achieve more generalized navigation in various environments. See below. Figure 6 A detailed example of how to operate SGR.

[0074] Policy updates can be performed event-by-event rather than step-by-step. For example, agent 210 in one or more embodiments may update the policy only in response to the completion of a navigation process, rather than modifying the policy at each step in the test environment, thereby improving the policy so that it adapts to the environment while maintaining consistency in the test environment.

[0075] Figure 3 Examples of policy update methods according to one or more embodiments are shown. (Refer to...) Figures 1A to 2 The provided description is also applicable Figure 3 .

[0076] Reference Figure 3 The input data may include test data X={X1, X2,..., X...} that is continuously streamed during the online test. n}. X n This may include instruction data (e.g., natural language instructions I) n ) and sensor data (e.g., image data).

[0077] Natural Language Commands I n The navigation goal to be performed by the agent 310 may include instructions (such as "move from the living room to the kitchen" or "go out the door and enter the hallway"). Natural language instructions may be provided as voice or text.

[0078] Initial sensor data This can be the starting point for the agent 310's navigation and can be provided, for example, as a 360° panoramic view. Sensor data can include omnidirectional visual information that the agent 310 can use to identify the surrounding environment within environment 130.

[0079] Sensor data can be processed by encoder 311, and command data can be processed by encoder 312. Encoders 311 and 312 can each convert the input data into a vector representation to transmit the input data to policy network 313. Policy network 313 can be an artificial neural network model that approximates the policy. Policy network 313 can be a neural network that can mathematically model the policy and learn the policy.

[0080] Policy Network 313 can be based on learned policy π θ To determine the action to be taken in the current state. For example, the policy network 313 may be based on a probability distribution such as Equation 1 below.

[0081] Equation 1:

[0082] In equation 1, s t It is the current state and may include sensor data converted by encoder 311, a t Actions are selected by the policy network 313 and may include actions such as movement or rotation of the agent 310.

[0083] When agent 310 is based on policy π θ When an action is performed, a navigation trajectory τ can be generated. The navigation trajectory τ can represent a series of states until navigation is completed and a set of actions performed by agent 310, and can be defined, for example, by the following equation 2.

[0084] Equation 2:

[0085] In Equation 2, τ can represent the entire navigation trajectory executed by agent 310, and T can represent the total number of steps executed by agent 310 during the navigation process.

[0086] For example, agent 310 may start from an initial state s0, select actions based on a policy, and continuously accumulate states and actions to form a navigation trajectory τ until the final target point is reached and / or navigation terminates.

[0087] The oracle 320 can evaluate the navigation results of the agent 310 and provide feedback F. For example, the feedback can be defined by Equation 3 below, depending on whether the target has been reached.

[0088] Equation 3:

[0089] Feedback can also be set as a distance-based, continuous reward value. For example, feedback can be represented by Equation 4 below (e.g., considering the final position s of agent 310). T With target point s The distance d(s) between T ,s )).

[0090] Equation 4:

[0091] The methods of providing feedback are not limited to Equations 3 and 4.

[0092] Policy updates can be performed on an event-by-event basis. For example, instead of updating the policy at each individual step, the policy can be updated based on feedback provided by oracle 320 in response to the completion of the navigation process (e.g., in response to the completion of an event).

[0093] For example, policy updates can be performed using a policy optimization function based on reinforcement learning (such as Equation 5 below).

[0094] Equation 5:

[0095] In equation 5, It can be used for strategies The objective function to be optimized is G. t It could be the expected reward at time t. It can be based on strategy Path sampling. For example, G t It can be represented by the following equation 6.

[0096] Equation 6:

[0097] In equation 6, R t It is the reward at time t and may include feedback F, and γ is a discount factor that can adjust the contribution of future rewards.

[0098] Policy updates are performed event-by-event, so the reward for each step can be 0, and F can be applied only in the last step. Therefore, the final reward function can be simplified to, for example, as shown in Equation 7 below.

[0099] Equation 7:

[0100] In equation 7, It is a reinforcing element, and can be a way for the agent 310 to update the policy based on feedback received in the final step.

[0101] For example, policy gradient It can be approximately represented by the following equation 8.

[0102] Equation 8:

[0103] The policy gradient can be a directional vector indicating how decisions made by the policy should be adjusted based on success / failure. In Equation 8, In state s t Choose action a t The probability is the derivative of the logarithm of the probability with respect to the policy parameter θ, and can be used to increase or decrease the probability of performing a predetermined action. For example, It is an indicator of how important the predetermined action is. For example, the indicator can be updated more significantly when the predetermined action is the correct action.

[0104] However, when learning is performed continuously on failed paths, there is a possibility that patterns of moving in the wrong direction may be reinforced. For example, blindly following gradients even in the event of failure may lead to getting stuck in a predetermined failure pattern.

[0105] Agent 310 can randomly reverse a portion of the updated gradient. Agent 310 can reverse a portion of the updated gradient and be guided to perform navigation in various ways. For example, even in the event of failure, agent 310 can not learn entirely in the opposite direction, but can be guided to randomly adjust one or more gradients to learn new patterns.

[0106] For example, as shown in Equation 9 below, agent 310 can use a gradient mask m to transform a portion of the gradient.

[0107] Equation 9:

[0108] m is a value sampled according to a predetermined probability distribution M, and can be used to invert or scale a portion of the gradient. M determines the random pattern for inverting the predetermined gradient. The probability p controls the rate at which the gradient is inverted. A portion of the gradient can be inverted with probability p, while the remaining gradient can retain its original direction with probability 1-p, but its magnitude can be adjusted. α represents the scaling factor that adjusts the magnitude of the inverted gradient. α determines the magnitude of the inverted gradient with probability p.

[0109] In summary, agent 310 can apply a gradient mask m, where a portion of the gradient can be inverted with probability p. The gradient is scaled by a factor of α, while the remainder of the gradient can be scaled with probability 1-p. This doubles the amount of time spent on ensuring consistency.

[0110] Finally, as shown in Equation 10 below, for example, agent 310 can apply a gradient mask m to the gradient and determine the gradient with the mask applied.

[0111] Equation 10:

[0112] In Equation 10, ⊙ represents element-wise multiplication (Hadamard product) and indicates that each gradient value is adjusted by the corresponding mask value m. See below for further details. Figure 6 Provide an example of a method that describes in detail a portion of the random gradient reversal.

[0113] As shown in Equation 11 below, for example, agent 310 can guide robust adaptive learning by randomly discarding parameter update processing during the processing of updating policy parameters.

[0114] Equation 11:

[0115] In Equation 11, agent 310 can be based on Bernoulli distribution (e.g., Bernoulli (1-p) generates a discard mask and applies it to the existing gradients to randomly update the parameters, thus effectively preventing the catastrophic forgetting problem of losing information in the existing learning domain. In Equation 11, This indicates the updated policy parameters. This indicates the policy parameter that will be updated.

[0116] Figure 4 Examples of providing binary feedback in a policy update method according to one or more embodiments are shown. (Refer to...) Figures 1A to 3 The provided descriptions are substantially the same and can be applied to [other applications]. Figure 4 .

[0117] The system can describe how to provide feedback based on events, where binary feedback can be provided based on whether a target point has been reached in response to an agent performing navigation according to a given policy. For example, the agent may not modify the policy while performing navigation and may update the policy by receiving feedback in response to the termination and / or completion of a navigation process (e.g., an event).

[0118] An agent can perform navigation based on input data. Input data may include natural language instructions and environmental information (e.g., sensor data), and can be processed in a policy network to determine the optimal action. The agent can perform navigation based on a policy and can receive feedback from an oracle when navigation is terminated and / or completed.

[0119] Figure 4 Examples of providing feedback in each of the two events 410 and 420 are shown. In the first event 410, when the agent reaches the target point, first feedback 411 may be provided. First feedback 411 may have a value of +1 (success), and the policy network can use this information to strengthen the policy. In the second event 420, when the agent does not reach the target point, second feedback 421 may be provided. Second feedback 421 may have a value of -1 (failure), and the policy network may be updated based on this to compensate for the navigation failure.

[0120] The policy network can operate based on previously learned policies, but can be adjusted to adapt itself by utilizing feedback collected in the testing environment. Thus, the methods and apparatus of one or more embodiments can improve the generalization ability of a model without it overfitting to a predetermined environment.

[0121] Figure 5 Examples of detailed instructions and corresponding events executed in a policy update method according to one or more embodiments are shown. (Refer to...) Figures 1A to 4 The provided descriptions are substantially the same and can be applied to [other applications]. Figure 5 .

[0122] Reference Figure 5 The system can describe a process in which an agent interprets given instructions based on a VLN model, performs navigation in the environment, and then receives feedback based on events.

[0123] Input data may include natural language instructions 510 and environmental information (sensor data). For example, instructions 510 (such as, “Go to the restaurant on the second floor and pull out the white chair closest to the living room”) may be provided. An agent receiving instructions 510 may perform the action of going to the restaurant on the second floor in the given environment and then pulling out the white chair closest to the living room.

[0124] An agent can use sensor data to determine its current location and determine a movement path based on a policy network. For example, an agent might perform the action of "climbing the stairs to the second floor, finding the chair closest to the living room in the dining room, and then pulling that chair." In response to navigation being terminated and / or completed, the oracle can evaluate the agent's performance and provide feedback.

[0125] When the agent executes the action exactly as instructed by instruction 510, the oracle provides a feedback value of +1 and an evaluation of "Correct!". For example, when the agent goes to a room other than the restaurant and / or chooses a chair other than the correct one, the oracle provides a feedback value of -1 and an evaluation of "Incorrect!". In this way, the agent can receive positive or negative feedback based on whether it has reached the target point and the result of the action, and can update the policy based on the feedback.

[0126] Figure 6 Examples of applying the SGR technique in a policy update method according to one or more embodiments are shown. (Refer to...) Figures 1A to 5 The provided descriptions are substantially the same and can be applied to [other applications]. Figure 6 .

[0127] Figure 6 A comparison is shown between the general gradient update method 610 and the method 620 applying SGR. The general gradient update method 610 can be the case without applying the SGR technique. In the general gradient update method 610, learning can be performed such that the gradient vector converges to the optimization objective (e.g., the minimum of the loss function). In the general gradient update method 610, the gradient of a single sample typically follows the direction of convergence, and when the predetermined pattern is repeated, there is a possibility that the gradient may fall into a local optimum during the convergence process.

[0128] Method 620 applying SGR can be a case of applying the SGR technique. In the SGR technique, one or more gradients can be reversed according to a predetermined probability p. In method 620 applying SGR, the reversed gradients 621 and 622 can be gradient vectors updated in the opposite direction to the original direction.

[0129] One or more embodiments of the SGR technique can prevent overfitting to a predetermined pattern and expand the navigation space by randomly reversing the direction of one or more gradients. For example, generalization performance may degrade when optimizing a policy trained on the same data distribution only in a predetermined direction. However, by randomly reversing one or more gradients, the methods and apparatus of one or more embodiments can achieve more diverse navigation during the learning process, which increases the likelihood that the policy will learn more generalized patterns.

[0130] Figure 7 Examples of policy update methods according to one or more embodiments are shown. (Refer to...) Figures 1A to 6 The provided descriptions are substantially the same and can be applied to [other applications]. Figure 7 .

[0131] although Figure 7Operations 710 to 730 may be performed in the order and manner shown therein, but without departing from the spirit and scope of the described embodiments, one or more of the operations may be performed in a different order, one or more of the operations may be omitted, two or more of the operations may be performed in parallel or simultaneously, and / or additional operations may be performed.

[0132] In Operation 710, based on the learned policy and the input data, the agent (e.g., Figure 1A The agent 110 can determine the agent's movement path. Input data may include instruction data (e.g., natural language instructions) and sensor data (e.g., image data). The movement path may include the navigation trajectory described above.

[0133] In operation 720, in response to completing a move, the agent receives feedback based on whether the target point has been reached. The agent can receive feedback from the oracle on an event-by-event basis.

[0134] Policy update methods are processes that adapt a pre-learned policy to the test-time. Therefore, policy update methods can be a way for an agent to receive feedback on the overall execution outcome in response to the end of an event, rather than predicting the correct answer at every moment (e.g., rather than predicting the correct answer when each step of the event is completed). Oracles can determine whether the agent has correctly performed navigation during this process and provide feedback (e.g., binary feedback). Oracles can operate based on feedback directly provided by humans, and / or AI-based systems can perform the evaluation automatically. For example, a large language model (LLM) can act as an oracle. When an agent performs navigation and requests evaluation from the LLM, the LLM can compare the natural language instructions with the movement outcome to determine success or failure. For this purpose, the LLM can be pre-provided with "correct answer information" or can infer the correct answer by utilizing environmental data.

[0135] An agent can receive positive feedback when it reaches a target point and negative feedback when it fails to reach the target point. For example, an oracle can provide a +1 feedback when the agent reaches the target point or a -1 feedback when the agent fails to reach the target point.

[0136] An agent can receive positive feedback by applying a first predefined weight when it reaches a target point, and negative feedback by applying a second predefined weight when it fails to reach the target point. For example, an oracle could provide +5 feedback when the agent reaches the target point, or -2 feedback when it fails. When feedback is provided as in the example above, the agent can receive a large reward upon reaching the target point, allowing it to learn successful policies more strongly. While learning continues even in the event of failure, the impact of success can be relatively large. For example, when feedback is provided as described above, the policy could be "prioritizing stable success over navigation."

[0137] An agent can receive continuous feedback based on the distance between its final position and the target point. For example, an oracle can set rewards based on the distance to the target rather than a simple 0 or 1 for success or failure. For instance, an oracle can set rewards to be negatively correlated with the distance to the target, causing it to set larger reward values ​​for shorter distances rather than longer distances. When feedback is provided as described above, the policy can be progressively improved. Therefore, providing feedback as in the example above allows for finer-grained policy updates compared to binary feedback.

[0138] In Operation 730, the agent can update the parameters of the learned policy based on feedback. Instead of training the policy during the training phase, the agent can update one or more parameters of the policy in real time using feedback during the inference phase.

[0139] The agent can update the gradient of the learned policy based on feedback, and update the parameters of the learned policy based on the updated gradient.

[0140] An agent can randomly reverse a portion of the updated gradient. The agent can determine whether to reverse the updated gradient based on predefined probabilities. When it determines to reverse the updated gradient, the agent can reverse the sign of the gradient. The agent can adjust the magnitude of the reversed gradient according to predefined coefficients.

[0141] The policy can be represented by a neural network, and the parameters of the policy can include all learnable parameters of the neural network (e.g., weights or biases). Therefore, an agent can update parameters including the weights of an artificial neural network model, which executes the learned policy. The artificial neural network model can be implemented as a reinforcement learning-based VLN model.

[0142] Figure 8 Examples of electronic devices according to one or more embodiments are shown. (Refer to...) Figures 1A to 7 The provided descriptions are substantially the same and can be applied to [other applications]. Figure 8 .

[0143] Reference Figure 8 The electronic device 800 may include a memory 810 (e.g., one or more memories) and a processor 830 (e.g., one or more processors). The electronic device 800 may be or may include one or both of the agents (e.g., agent 110, agent 210 and / or agent 310) and oracles (e.g., oracle 120, oracle 220 and / or oracle 320) as described above.

[0144] The memory 810 may store instructions (or programs) that can be executed by the processor 830. For example, the instructions may include instructions for performing operations of the processor 830 and / or operations of each component of the processor 830.

[0145] The memory 810 can be implemented as a volatile memory device or a non-volatile memory device.

[0146] Volatile memory devices can be implemented as dynamic random access memory (DRAM), static RAM (SRAM), thyristor RAM (T-RAM), zero-capacitor RAM (Z-RAM), and / or dual-transistor RAM (TTRAM).

[0147] Non-volatile memory devices can be implemented as electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic RAM (MRAM), spin-transfer torque RAM (STT-MRAM), conductive bridged RAM (CBRAM), ferroelectric RAM (FeRAM), phase-change RAM (PRAM), resistive RAM (RRAM), nanotube RRAM, polymer RAM (PoRAM), nanofloating gate memory (NFGM), holographic memory, molecular electronic memory devices, and / or insulator resistance change memory.

[0148] Processor 830 can process data stored in memory 810. Processor 830 can execute computer-readable code (e.g., software) and instructions triggered by processor 830 stored in memory 810. For example, memory 810 may be or may include a non-transitory computer-readable storage medium storing instructions that, when executed by processor 830, configure processor 830 to perform actions described herein. Figures 1A to 8 Any, any combination of, or all of the disclosed operations and / or methods.

[0149] The processor 830 may be a hardware-implemented data processing device having circuitry physically configured to perform a desired operation. The desired operation may include, for example, code or instructions in a program.

[0150] For example, hardware-implemented data processing devices may include microprocessors, central processing units (CPUs), processor cores, multi-core processors, multiprocessors, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0151] Processor 830 can determine the agent's movement path based on input data according to the learned policy, obtain feedback based on whether the target point has been reached in response to the agent completing the movement, and update the parameters of the learned policy based on the feedback. Processor 830 can perform reference in essentially the same way. Figures 1A to 7 The described operation. Therefore, its detailed description is omitted.

[0152] The encoders, electronic devices, memory, processors, encoders 311 and 312, electronic devices 800, memory 810, and processors 830 described herein (including those related to...) Figures 1A to 8The description of the hardware components (as described above) is implemented or represents a hardware component. Examples of hardware components that can be used to perform the operations described in this application, as appropriate, include, as appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units (ALUs), digital signal processors (DSPs), microcomputers, programmable logic controllers, field-programmable gate arrays (FPGAs), programmable logic arrays (PLUs), microprocessors, or any other means or combination of means configured to respond to and execute instructions (e.g., code or encoding) in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by a processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) to perform the operations described in this application. Hardware components can also access, manipulate, process, create, and store data in response to the execution of instructions or software. For brevity, the singular terms "processor" or "computer" may be used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both, and therefore while some references may be made to a single processor or computer, such references are also intended to refer to multiple processors or computers. For example, a single hardware component, or two or more hardware components, may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. As described above, or in addition to the above description, the example hardware components may have any one or more different processing configurations, examples of which include: a single processor, a discrete processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.Therefore, reference to a processor herein refers to a processing circuit system (e.g., a circuit system including one or more processing element circuits). One or more processors including a processing circuit system also refers to each processor including a processing circuit system, and some or all of one or more processors including the same processing circuit system. Furthermore, one or more processors and one or more controllers, as non-limiting examples, do not represent human processing or human control, but rather represent hardware components as described herein as non-limiting examples.

[0153] Performing the operations described in this application Figures 1A to 8 Shown and about Figures 1A to 8 The methods discussed are executed by computing hardware (e.g., by one or more processors or a computer), which is implemented as described above to implement instructions (e.g., computer or processor / processing device readable instructions) or software to perform the operations performed by the methods described in this application. For example, a single operation, or two or more operations, may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may perform a single operation, or two or more operations. References to processors, or one or more processors, configured to perform two or more operations as non-limiting examples indicate that processors, or two or more processors, are configured to jointly perform all of the two or more operations, and that two or more processors respectively perform any corresponding operation of the two or more operations (e.g., corresponding one or more processors are configured to perform each of the two or more operations, or any corresponding combination of one or more processors is configured to perform any corresponding combination of the two or more operations). Similarly, a reference to a processor-implemented method is a reference to a method executed by one or more processors or other processing or computing hardware of a device or system.

[0154] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above may be written as computer programs, code segments, or other executable instructions or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software may be written in any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding description herein, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0155] Instructions or software used to control computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, may be recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media, and are therefore not the signal itself. Thus, references to storage media herein refer to storage media hardware and not to transient media, not the signal itself. As described above, or in addition to the above description, examples of non-transitory computer-readable storage media include any one or more of the following: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage devices, hard disk drives (HDDs), solid-state drives (SSDs), card storage devices (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state drives, and / or any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner, and to provide instructions or software and any associated data, data files, and data structures to one or more processors or computers, such that one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system, such that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0156] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered descriptively only and not for limiting purposes. The description of features or aspects in each example should be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents.

[0157] Therefore, in addition to the above and all the accompanying drawings, the scope of the disclosure also includes the claims and their equivalents (i.e., all variations within the scope of the claims and their equivalents should be interpreted as included in the disclosure).

Claims

1. A method for policy updating, comprising: Based on the learned strategy, the agent's movement path is determined using the input data; In response to the agent completing a movement, feedback is obtained based on whether the target point has been reached; as well as Based on the feedback, the parameters of the learning strategy are updated.

2. The method according to claim 1, wherein, The steps to obtain the feedback include: obtaining the feedback for each event.

3. The method according to claim 1, wherein, The steps for updating the parameters include: Based on the feedback, the gradient of the learned strategy is updated; and Based on the updated gradient, the parameters of the learned policy are updated.

4. The method according to claim 3, wherein, The step of updating the parameters also includes randomly reversing a portion of the updated gradient.

5. The method according to claim 4, wherein, The steps to reverse a portion of the updated gradient include: Based on predefined probability values, determine whether to reverse the updated gradient; and In response to determining the inverted updated gradient, the sign of the updated gradient is inverted.

6. The method according to claim 5, wherein, The steps to reverse the sign of the updated gradient include: adjusting the magnitude of the reversed gradient according to predefined coefficients.

7. The method according to claim 1, wherein, The step of updating the parameters includes updating parameters that include the weights of the artificial neural network model, which is configured to perform a learning strategy.

8. The method according to claim 7, wherein, The artificial neural network model was implemented as a visual language navigation model based on reinforcement learning.

9. The method according to claim 1, wherein, The steps to obtain the feedback include: In response to the agent reaching the target point, it receives positive feedback; and The agent receives negative feedback when it fails to reach the target point.

10. The method according to claim 9, wherein, The steps for updating the parameters of a learning strategy include: In response to receiving positive feedback, the parameters of the learned policy are to apply the first predefined weights. In response to receiving negative feedback, a second predefined weight is applied to the parameters of the learned policy.

11. The method according to claim 9, wherein, The steps for obtaining the feedback include: obtaining continuous value feedback based on the distance between the agent's final position and the target point.

12. The method according to any one of claims 1 to 11, further comprising: Whether to generate the feedback is determined based on whether the agent has completed the event. The event comprises a series of steps, each of which includes a corresponding action performed by the agent along the movement path. The step of obtaining the feedback includes: in response to determining that the feedback is to be generated, generating the feedback based on a comparison between the last step in the series of steps and the target point.

13. An electronic device comprising: One or more processors, including processing circuitry; as well as The memory includes one or more storage media storing instructions that, when executed individually or jointly by the one or more processors, cause the one or more processors to: Based on the learned strategy, the agent's movement path is determined using the input data; In response to the agent completing a movement, feedback is obtained based on whether the target point has been reached; as well as Based on the feedback, the parameters of the learning strategy are updated.

14. The electronic device according to claim 13, wherein, In order to obtain the feedback, the execution of the instructions causes the one or more processors to obtain the feedback for each event.

15. The electronic device of claim 13, wherein, In order to update the parameters, the execution of the instruction causes the one or more processors to: Based on the feedback, the gradient of the learned strategy is updated; and Based on the updated gradient, the parameters of the learned policy are updated.

16. The electronic device according to claim 15, wherein, In order to update the parameters, the execution of the instruction causes the one or more processors to randomly reverse a portion of the updated gradient.

17. The electronic device of claim 16, wherein, In order to reverse a portion of the updated gradient, the execution of the instruction causes the one or more processors to: Based on predefined probability values, determine whether to reverse the updated gradient; and In response to determining the inverted updated gradient, the sign of the updated gradient is inverted.

18. The electronic device according to claim 13, wherein, In order to reverse the sign of the updated gradient, the execution of the instruction causes the one or more processors to update parameters including the weights of an artificial neural network model configured to perform a learning strategy.

19. The electronic device of claim 13, wherein, In order to obtain the feedback, the execution of the instruction causes the one or more processors to: In response to the agent reaching the target point, it receives positive feedback; and The agent receives negative feedback when it fails to reach the target point.

20. A method for policy updating, comprising: Whether to generate feedback is determined based on whether the agent has completed an event, wherein the event comprises a series of steps, and each of the series of steps comprises a corresponding action performed by the agent; In response to determining the generation of feedback, feedback is generated based on a comparison between the final step in the series of steps and the target point; and Based on the feedback, the parameters of the learning strategy are updated.

Images