A method and system for safe post-training of embodied multi-modal large models

Abstract

The application discloses a method and system for safe post-training of embodied multimodal large models. The method includes introducing a world model as a sandbox for safe training. After the embodied multimodal large model generates an initial action, the initial action is input into the world model to predict future images. A multimodal safety discriminator is used to assess the safety risk of the future images. If the initial action is determined to be dangerous, the initial action is automatically corrected based on a risk description, an alternative action is generated, and the alternative action is returned to the world model for verification. This process is repeated until the corrected alternative action is determined to be safe. The dangerous initial action and the safe alternative action are used to construct a training data pair, and an embodied multimodal large model is updated through a reinforcement learning algorithm. Thus, without causing harm through real physical interaction, the model's ability to predict potential physical risks in complex scenarios and its ability to refuse or avoid such risks are improved, and the behavior compliance and safety of the model in an open instruction space are achieved.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to the fields of large model and robot control, and particularly to a secure post-training method and system for embodied multimodal large models. Background Technology

[0002] With the development of embodied intelligence, embodied multimodal large models are widely used in the field of robot control. These models not only need to understand text instructions, but also process visual input and output physical actions. However, in practical deployment, the safety of the model is crucial. Traditional training methods often focus on task success rate, such as "pick up the cup," while neglecting safety in the process, such as "don't knock over the vase next to you."

[0003] Currently, specific multimodal large model control schemes are mainly divided into two categories: Reinforcement Learning from Human Feedback (RLHF) collects human preference data for model outputs (action descriptions or trajectories), trains a reward model, and uses the proximal policy optimization (PPO) algorithm to optimize the policy. Its drawback is that, in embodied scenarios, obtaining real-world robot violation data (such as crashing into objects or accidentally injuring people) is extremely costly and dangerous. It's unacceptable to have the robot perform 10,000 dangerous actions to collect negative samples just to train a "safe" model.

[0004] Rule-based or constraint-based reinforcement learning (Safe Reinforcement Learning, or Safe RL) adds predefined constraints (such as joint angle limits or obstacle avoidance distances) to the loss function. Its drawback is poor generalization ability and difficulty in handling complex semantic safety issues within an "open instruction space." For example, the instruction "pour water onto the computer" may conform to joint constraints at the physical motion level, but it is semantically unsafe; traditional constraints cannot recognize this high-level risk.

[0005] In summary, existing technologies cannot enable embodied models to autonomously learn complex causal safety risks in the physical world without performing dangerous actions. As a result, when faced with complex and unseen scenarios, the models lack the ability to predict and avoid future risks. Summary of the Invention

[0006] This application provides a secure post-training method and system for embodied multimodal large models to address how to improve the model's ability to predict potential physical risks in complex scenarios and its ability to refuse / avoid responses without causing harm through real physical interactions during the post-training stage, thereby achieving compliance and security of the model's behavior in an open instruction space.

[0007] The technical solution is as follows: Firstly, a secure post-training method for embodied multimodal large models is provided, including: The user command is input into the embodied multimodal large model, and based on the acquired current environment image and the user command, the initial action corresponding to the user command is output. The initial action and the current environment image are input into a pre-trained image generative world model for prediction, and the output is a future image that will occur when the initial action is performed. The future image and preset safety prompts are input into a multimodal safety discriminator for analysis, and the output is a safety probability and risk description characterizing whether the initial action is safe or not. The safety probability is input into the reward generation engine for judgment. If the safety probability is lower than a preset probability threshold, the initial action is determined to be dangerous; if the safety probability is not lower than the preset probability threshold, the initial action is determined to be safe. The trigger action correction engine performs safety corrections on the initial dangerous action based on the risk description to generate a counterfactual action, which is then returned to the image-generating world model for verification. This process is repeated until the reward generation engine determines that the corrected counterfactual action is safe. Training data is constructed by pairing safe counterfactual actions and dangerous initial actions, and the embodied multimodal large model is updated by combining reinforcement learning algorithms.

[0008] In one possible implementation, the action correction engine performs safety corrections on the initial dangerous action based on the risk description to generate a counterfactual action, specifically including: The action correction engine extracts keywords that characterize danger from the risk description, and fine-tunes or logically corrects the initial action that poses a danger based on the keywords to generate counterfactual actions. The fine-tuning includes correcting at least one of the motion path, motion height, and motion speed of the initial action; the logical correction includes changing the motion path of the initial action.

[0009] In one possible implementation, training data is constructed based on pairs of safe counterfactual actions and dangerous initial actions, and this is combined with reinforcement learning to update the embodied multimodal large model, specifically including: Preference pairs are constructed based on safe counterfactual actions and dangerous initial actions: (I_t, T, a'_t, a_t), where I_t represents the current environment image, T represents the user command, a'_t represents the safe counterfactual action, a_t represents the dangerous initial action, and a'_t corresponds to high reward and a_t corresponds to high penalty. Using the aforementioned preference pairs, the parameters of the embodied multimodal large model are updated using reinforcement learning algorithms that employ direct preference optimization or proximal policy optimization, so that it generates actions that conform to physical safety constraints.

[0010] In one possible implementation, the multimodal security discriminator is a semantic-level physical security discriminator; then, the future image and preset security warning words are input into the multimodal security discriminator for analysis, specifically including: The semantic-level physical safety discriminator receives future images and preset safety warning words, and combines them with common sense about physics to make causal inferences about the potential consequences of future images as long-term tasks; wherein, the common sense about physics includes: easy to spill, fragile, conductive, and high temperature.

[0011] In one possible implementation, the preset probability threshold ranges from 0.4 to 0.8.

[0012] In one possible implementation, when the initial action is determined to be safe, the method further includes: The robot corresponding to the embodied multimodal large model directly executes the initial action.

[0013] Secondly, a secure post-training system for embodied multimodal large models is provided, comprising: an embodied multimodal large model, an image-generative world model, a multimodal secure discriminator, a reward generation engine, an action correction engine, and a reinforcement learning optimizer; wherein, The embodied multimodal large model is used to receive user instructions and acquire current environmental images, make predictions based on the user instructions and the current environmental images, and output the initial action corresponding to the user instructions. The image-generative world model is used to receive the initial action and the current environment image, make predictions, and output a future image that will occur when the initial action is performed. The multimodal security discriminator is used to receive the future image and preset security prompt words, analyze them, and output a security probability and risk description that characterizes whether the initial action is safe or not. The reward generation engine is used to receive the safety probability and make a judgment. If the safety probability is lower than a preset probability threshold, the initial action is determined to be dangerous; if the safety probability is not lower than the preset probability threshold, the initial action is determined to be safe. The action correction engine is used to make safety corrections to the initial dangerous action based on the risk description, so as to generate a counterfactual action and return it to the image-generated world model for verification. This process is repeated until the reward generation engine determines that the corrected counterfactual action is safe. The reinforcement learning optimizer is used to construct pairs of training data based on safe counterfactual actions and dangerous initial actions, and to update the embodied multimodal large model in conjunction with reinforcement learning algorithms.

[0014] Thirdly, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described above and any possible implementations.

[0015] Fourthly, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the aspects described above and any possible implementation thereof.

[0016] Fifthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the aspects and any possible implementations described above.

[0017] As can be seen from the above technical solution, this application introduces a world model as a sandbox for security training. When the embodied multimodal large model generates an initial action, it is input into the world model to predict future images. Then, a multimodal security discriminator assesses the security risks of the future images. If the initial action is determined to be dangerous, it is automatically corrected based on the risk description, generating a counterfactual action, which is then returned to the world model for verification. This process is repeated until the corrected counterfactual action is determined to be safe. Finally, the dangerous initial action and the safe counterfactual action are used to construct training data pairs, and the embodied multimodal large model is updated through reinforcement learning algorithms. Thus, without causing harm through real physical interaction, the model's ability to predict potential physical risks in complex scenarios and its ability to refuse / avoid responses are improved, thereby achieving compliance and security of the model's behavior in an open instruction space.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the structure of the embodied intelligent agent involved in the embodiments of this application; Figure 2 This is a schematic diagram illustrating the steps of a secure post-training method for embodied multimodal large models provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the secure post-training method for embodied multimodal large models provided in this application embodiment; Figure 4 This is a structural block diagram of the embodied multimodal large model secure post-training system provided in the embodiments of this application; Figure 5 This is a block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0021] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These embodiments should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] Obviously, the described embodiments are only some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0023] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0024] Given that existing technologies cannot enable embodied models to autonomously learn complex causal safety risks in the physical world without performing dangerous actions, the models lack the ability to predict and avoid future risks when facing unseen complex scenarios. This application proposes a safety post-training scheme for embodied multimodal large models based on counterfactual inference from a world model. The main inventive concept is as follows: First, a world model is introduced as a sandbox for safety training. When the embodied multimodal large model generates an initial action, it is input into the world model to predict future images. Then, a multimodal safety discriminator performs a safety risk assessment on the future images. If the initial action is determined to be dangerous, it is automatically corrected based on the risk description, generating a counterfactual action, which is then returned to the world model for verification. This process is repeated until the corrected counterfactual action is determined to be safe. Finally, dangerous initial actions and safe counterfactual actions are used to construct training data pairs, and the embodied multimodal large model is updated through a reinforcement learning algorithm. Thus, without causing harm through real physical interaction, the model's ability to predict potential physical risks in complex scenarios and its ability to refuse or avoid responses are improved, thereby achieving compliance and security of the model's behavior in an open instruction space.

[0025] Reference Figure 1 The diagram shown is a structural schematic of the embodied intelligent agent involved in this application embodiment. The embodied intelligent agent may include: an embodied robot body 101 and an embodied multimodal large-scale model safe post-training system 102 integrated on the embodied robot body 101. The embodied robot body 101 may include: a mechanical body, a robotic arm, a drive and transmission device, a sensor system, etc. The specific module structure of the embodied multimodal large-scale model safe post-training system 102 can be found in [reference needed]. Figure 4 As shown.

[0026] Reference Figure 2 The diagram shown is a step-by-step illustration of a secure post-training method for embodied multimodal large models provided in an embodiment of this application.

[0027] This method may specifically include the following steps: Step 202: Input the user command into the embodied multimodal large model, and output the initial action corresponding to the user command based on the collected current environment image and the user command.

[0028] In this application, the embodied multimodal large model (ELM / EMLM) can be a robot "brain" that integrates multiple modalities such as text, vision, audio, and touch, providing intelligent agents with a complete embodied intelligence solution model from perception and reasoning to execution. For example, it can be applied to embodied intelligent robots.

[0029] Based on the image acquisition devices such as cameras deployed on the embodied intelligent robot, images of the current environment in which the embodied robot is located can be collected. It should be understood that the images here can be videos, video clips or pictures.

[0030] The user instructions can be machine instructions that the embodied intelligent robot can recognize, generated in response to user input via voice or text. These user instructions can be requests or instructions for the embodied intelligent robot to perform some household services or to perform other assistance operations in various scenarios, such as "open the door for me," "get me a glass of water," or "turn on the machine for me."

[0031] The embodied multimodal large model described here can perform analysis, perception, and prediction based on user input commands and the current environmental image, and then output the initial action required to complete the user command. It should be understood that the initial action here can be a single action or a set of actions, and this application does not limit it in this way.

[0032] In this application, the initial action can be a representation of an action intention, such as a specific end effector trajectory or macro action instruction.

[0033] Step 204: Input the initial action and the current environment image into a pre-trained image generative world model for prediction, and output the future image that will occur when the initial action is performed.

[0034] It should be noted that an image generative world model can be a world model used to predict images that may occur in the future, that is, inputting necessary parameters and outputting future images.

[0035] Unlike existing technologies, in this application, after the embodied multimodal large model outputs an initial action, it does not execute it immediately. Instead, it inputs the initial action into a pre-trained image-generative world model, along with the current environment image. The image-generative world model analyzes and predicts based on the input initial action and the current environment image, outputting a future image that would occur if the initial action were executed.

[0036] One alternative is that the future image could be a future video clip.

[0037] Step 206: Input the future image and preset safety prompt words into the multimodal safety discriminator for analysis, and output the safety probability and risk description characterizing whether the initial action is safe or not.

[0038] In this application, the preset safety prompts can be descriptive phrases used to alert users to potential risks; these can be words or sentences. For example, "Check for damaged objects or improper operation" or "Have there been any collisions?". These preset safety prompts can be derived from user experience or customized according to user habits.

[0039] A multimodal safety discriminator can employ common architectures, such as convolutional neural networks or attention mechanisms. It receives real data or data generated by a generator as input and outputs a probability value between 0 and 1, representing the likelihood that the sample is real data. It can also output a risk description, such as "liquid spilled on electronic device." Thus, a vision-language-based safety discriminator evaluates future videos predicted by a world model (i.e., an image-generative world model). If the prediction indicates a risk (e.g., broken object, liquid spill), a high penalty is applied; if the prediction indicates no risk, a high reward is given.

[0040] The multimodal security discriminator is a semantic-level physical security discriminator; the semantic-level physical security discriminator receives future images and preset security warning words, and combines physical common sense to make causal inferences about the potential consequences of future images as long-term tasks; wherein, the physical common sense includes: easily spilled, fragile, conductive, and high temperature.

[0041] Step 208: Input the safety probability into the reward generation engine for judgment. If the safety probability is lower than a preset probability threshold, the initial action is determined to be dangerous; if the safety probability is not lower than the preset probability threshold, the initial action is determined to be safe.

[0042] In actual control and processing, different preset probability thresholds can be set according to different application scenarios. The range of the preset probability threshold is 0.4-0.8.

[0043] For example, a preset probability threshold of 0.7 can be set. Then, the reward generation engine determines the safety probability. If the safety probability is 0.1, it is less than the preset probability threshold of 0.7, and the initial action is considered dangerous. If the safety probability is 0.9, it is greater than the preset probability threshold of 0.7, and the initial action is considered safe.

[0044] Step 210: The action correction engine performs safety corrections on the initial dangerous action based on the risk description to generate a counterfactual action, and returns it to the image-generating world model for verification. This process is repeated until the reward generation engine determines that the corrected counterfactual action is safe.

[0045] If an initial action is deemed dangerous, it needs to be corrected. Specifically, an action correction engine can be used to make safe positive corrections to the dangerous initial action based on the risk description, generating a counterfactual action. This counterfactual action is then returned to the image-generative world model for verification. This process is repeated until a prediction result that both completes the task and is safe is obtained.

[0046] Optionally, when the action correction engine performs safety corrections on an initial action that poses a danger based on the risk description to generate a counterfactual action, it can specifically extract keywords representing danger from the risk description and fine-tune or logically correct the initial action that poses a danger based on the keywords to generate a counterfactual action; wherein, the fine-tuning includes correcting at least one of the action path, action height, and action speed of the initial action; the logical correction includes changing the action path of the initial action.

[0047] It should be noted that the keywords in this application can be words that may cause obvious accidents or dangers, such as "collision," "splashing," or "shaking." These are only examples and are not intended to limit the scope of the application.

[0048] One feasible solution is that, when the initial action is determined to be safe, the method can also trigger the robot corresponding to the embodied multimodal large model to directly execute the initial action.

[0049] Step 212: Construct pairs of training data based on safe counterfactual actions and dangerous initial actions, and update the embodied multimodal large model using reinforcement learning algorithms.

[0050] Optionally, training data is constructed based on pairs of safe counterfactual actions and dangerous initial actions, and this data is used to update the embodied multimodal large model using reinforcement learning. Specifically, preference pairs (I_t, T, a'_t, a_t) can be constructed based on safe counterfactual actions and dangerous initial actions. Here, I_t represents the current environment image, T represents the user instruction, a'_t represents the safe counterfactual action, and a_t represents the dangerous initial action. a'_t corresponds to a high reward, and a_t corresponds to a high penalty. Using these preference pairs, reinforcement learning algorithms employing direct preference optimization or proximal policy optimization are used to update the parameters of the embodied multimodal large model, enabling it to generate actions that conform to physical safety constraints. Here, high penalty and high reward typically refer to the numerical values ​​set in the reward function during reinforcement learning. High reward encourages specific behaviors: the larger the value, the more likely the agent is to repeat the action. High penalty suppresses dangerous or erroneous behaviors: for example, if liquid spills onto the computer while moving a cup, a score of -10 is awarded.

[0051] The embodied multimodal large model is updated by pairing "dangerous initial action - high penalty" and "safe counterfactual action - high reward" as training data and updating them through reinforcement learning. In this way, safety is learned by "imagining" the consequences, realizing the principle of "learning from mistakes (in imagination) and gaining wisdom (in reality)".

[0052] The updated embodied multimodal large model possesses strong semantic understanding capabilities, enabling it to accurately and quickly distinguish between dangerous and safe situations. For example, it can understand that "water" is dangerous to a "computer" but safe for a "cup," and it can handle complex safety constraints based on semantic and physical attributes. Consequently, it can execute more precise and secure behavioral services upon receiving subsequent tasks.

[0053] The above technical solution introduces a generative world model as an environment simulator into the reinforcement learning training loop. This model predicts future videos before actions are executed and performs safety assessments based on the predictions, rather than on the current state. Simultaneously, a semantic-level physical safety discriminator is introduced. This discriminator receives the predicted videos generated by the world model and, combined with physical common sense (such as fragility, conductivity, and high temperature), makes causal inferences about the potential consequences of long-term tasks, serving as a negative feedback source for the reward function. When a predicted action is determined to be unsafe, an optimization algorithm or large model prompts are automatically used to generate a corrected safe action a'. The (safe action a', unsafe action a) are then constructed as paired preference data for direct preference optimization (DPO). Thus, without causing harm through real physical interaction, the model's ability to predict potential physical risks in complex scenarios and its ability to refuse / avoid responses are improved, thereby achieving behavioral compliance and safety in an open instruction space.

[0054] Reference Figure 3 The diagram shown is a flowchart illustrating the secure post-training method for embodied multimodal large models provided in this application embodiment.

[0055] like Figure 3 As shown, taking a home service scenario as an example: a home service robot (wheeled mobility + robotic arm). Task: The user's instruction is "Bring me the hot coffee on the table," while a laptop is also on the table. Potential risk: The robot plans a straight path directly; the coffee cup passes over the laptop, posing a risk of spilling and damaging the computer (a combined semantic and physical risk).

[0056] Specifically, this training method can be divided into the following parts: perception phase, initial decision-making, virtual simulation, security judgment, counterfactual correction and re-simulation, and model update. Among them: Perception Phase: The user inputs the command "Give me hot coffee." This user command triggers the robot to capture an image of the desktop via its camera, including the coffee cup and the computer.

[0057] Initial decision: The embodied multimodal large model calculates a shortest path action a_{orig} based on the currently received parameters: user instructions and desktop image: grab in a straight line and quickly translate over the computer.

[0058] Virtual simulation: The system calls the world model and inputs the current frame I and the action a_{orig}. Output video V_{pred}: The video shows a coffee cup shaking during movement, with a small amount of liquid spilling onto the computer keyboard.

[0059] Safety Judgment: The multimodal safety discriminator analyzes and predicts video V_{pred}, identifying contact between "liquid" and "electronic device". This is judged as "unsafe", with a safety probability score of 0.1; risk description: "Liquid spilled onto electronic device". The preset probability threshold here can be 0.6.

[0060] Counterfactual Correction and Re-analysis: Based on the keyword "splashing," the action correction engine generates a corrective action, namely the counterfactual action a_{fix}: increasing the lifting height, avoiding the computer area, and reducing the movement speed by 30%. The world model re-analyzes the counterfactual action a_{fix}, outputting video V'_{pred}: showing the coffee cup moving smoothly without passing over the computer. The multimodal safety discriminator determines it as "safe," with a safety probability score of 0.95. At this point, the risk description can be: no danger.

[0061] It should be understood that counterfactual actions can be derived through multiple revisions and deductions until the final counterfactual action is deemed safe and its probability score is higher than the probability threshold. Furthermore, in this application's scheme, Figure 3 The safe branches shown in the diagram generally only appear after revision and re-analysis. If the initial determination is safe, the safe action can be executed directly.

[0062] Model Update: Next, training samples are constructed using safe counterfactual actions: Input=[image, instruction], Winner=[a_{fix}], Loser=[a_{orig}]. A loss function, such as the DPO loss, is calculated: the loss is calculated and the policy is updated based on the higher probability of the safe action a_{fix} compared to the unsafe action a_{orig}. Then, backpropagation updates the model, enabling it to learn the high-level semantic safety logic that "when electronic devices are present, liquids must be passed around." Consequently, in subsequent similar tasks, accurate and safe behavioral decisions can be made and executed.

[0063] As can be seen, this application's solution transfers physical risks to the computational domain by using a world model for trial and error in the "imagination." The robot can learn to avoid danger without performing a single dangerous action in the real world, thus significantly reducing training risks. Simultaneously, leveraging the semantic understanding capabilities of a multimodal large model, it can handle complex safety constraints based on semantic and physical attributes, improving the controllability of complex operations in open spaces. Furthermore, considering that data generation is parallel and digital, unaffected by the physical robot's movement speed, the generation of counterfactual data is equivalent to synthesizing high-quality "expert demonstrations," accelerating model convergence and significantly improving training efficiency.

[0064] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0065] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0066] Figure 4 This paper illustrates a structural block diagram of an embodiment of a secure post-training system for embodied multimodal large models provided in this application, as shown below. Figure 4As shown. The embodied multimodal large model safety post-training system 400 of this embodiment may include an embodied multimodal large model 401, an image-generative world model 402, a multimodal safety discriminator 403, a reward generation engine 404, an action correction engine 405, and a reinforcement learning optimizer 406. The embodied multimodal large model 401 is used to receive user instructions and acquire current environmental images, perform predictions based on the user instructions and the current environmental images, and output an initial action corresponding to the user instructions. The image-generative world model 402 is used to receive the initial action and the current environmental image, perform predictions, and output a future image that will occur when the initial action is performed. The multimodal safety discriminator 403 is used to receive the future image and preset safety warning words, perform analysis, and output a safety probability and risk description characterizing whether the initial action is safe or not. The reward generation engine 404 is used to receive the safety probability and make a judgment; if the safety probability is lower than a preset probability threshold, the initial action is determined to be dangerous; if the safety probability is not lower than the preset probability threshold, the initial action is determined to be safe. The action correction engine 405 is used to make safety corrections to the initial action that is dangerous based on the risk description, so as to generate counterfactual actions and return them to the image generative world model for verification. This process is repeated until the reward generation engine determines that the corrected counterfactual action is safe. The reinforcement learning optimizer 406 is used to construct pairs of training data based on safe counterfactual actions and dangerous initial actions, and update the embodied multimodal large model 401 in combination with reinforcement learning algorithms.

[0067] Optionally, in one possible implementation of this embodiment, when the motion correction engine 405 performs safety correction on an initial motion that poses a danger based on the risk description to generate a counterfactual motion, it is specifically used to extract keywords representing danger from the risk description, and to fine-tune or logically correct the initial motion that poses a danger based on the keywords to generate a counterfactual motion; wherein, the fine-tuning includes correcting at least one of the motion path, motion height, and motion speed of the initial motion; the logical correction includes changing the motion path of the initial motion.

[0068] Optionally, in one possible implementation of this embodiment, when the reinforcement learning optimizer 406 constructs training data based on safe counterfactual actions and dangerous initial actions, and updates the embodied multimodal large model using reinforcement learning, it specifically constructs preference pairs based on safe counterfactual actions and dangerous initial actions: (I_t, T, a'_t, a_t), where I_t represents the current environment image, T represents the user instruction, a'_t represents the safe counterfactual action, a_t represents the dangerous initial action, and a'_t corresponds to a high reward and a_t corresponds to a high penalty; using the preference pairs, the parameters of the embodied multimodal large model are updated using a reinforcement learning algorithm with direct preference optimization or proximal policy optimization, so that it generates actions that conform to physical safety constraints.

[0069] Optionally, in one possible implementation of this embodiment, the multimodal security discriminator 403 is a semantic-level physical security discriminator; then the multimodal security discriminator 403 is specifically used to receive future images and preset security prompts, and combine physical common sense to make causal inferences about the potential consequences of future images as long-term tasks; wherein, the physical common sense includes: easily spilled, fragile, conductive, and high temperature.

[0070] Optionally, in one possible implementation of this embodiment, the preset probability threshold ranges from 0.4 to 0.8.

[0071] Optionally, in one possible implementation of this embodiment, when the initial action is determined to be safe, the reward generation engine 404 is further used to trigger the robot corresponding to the embodied multimodal large model to directly execute the initial action.

[0072] In this embodiment, a world model can be introduced as a sandbox for safety training. When the embodied multimodal large model generates an initial action, it is input into the world model to predict future images. Then, a multimodal safety discriminator assesses the safety risks of the future images. If the initial action is deemed dangerous, it is automatically corrected based on the risk description, generating a counterfactual action, which is then returned to the world model for verification. This process is repeated until the corrected counterfactual action is deemed safe. Finally, dangerous initial actions and safe counterfactual actions are used to construct training data pairs, and the embodied multimodal large model is updated using reinforcement learning algorithms. Thus, without causing harm through real physical interaction, the model's ability to predict potential physical risks in complex scenarios and its ability to refuse / avoid responses are improved, thereby achieving compliance and safety of the model's behavior in an open instruction space.

[0073] One embodiment of this application provides a computer-readable storage medium storing at least one instruction that is loaded and executed by a processor to implement the embodied multimodal large model secure post-training method as described above.

[0074] One embodiment of this application provides an electronic device including a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the embodied multimodal large model secure post-training method as described above.

[0075] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0076] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0077] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0078] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0079] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the embodied multimodal large model secure post-training method. For example, in some embodiments, the embodied multimodal large model secure post-training method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the embodied multimodal large model secure post-training method described above can be performed. Alternatively, in other embodiments, computing unit 501 may be configured by any other suitable means (e.g., by means of firmware) to perform an embodied multimodal large model secure post-training method.

[0080] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, at least one input device, and at least one output device.

[0081] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0082] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0083] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0084] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0085] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0086] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0087] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A secure post-training method for embodied multimodal large models, characterized in that, include: The user command is input into the embodied multimodal large model, and based on the acquired current environment image and the user command, the initial action corresponding to the user command is output. The initial action and the current environment image are input into a pre-trained image generative world model for prediction, and the output is a future image that will occur when the initial action is performed. The future image and preset safety prompts are input into a multimodal safety discriminator for analysis, and the output is a safety probability and risk description characterizing whether the initial action is safe or not. The safety probability is input into the reward generation engine for judgment. If the safety probability is lower than a preset probability threshold, the initial action is determined to be dangerous. If the safety probability is not lower than a preset probability threshold, then the initial action is determined to be safe. The trigger action correction engine performs safety corrections on the initial dangerous action based on the risk description to generate a counterfactual action, which is then returned to the image-generating world model for verification. This process is repeated until the reward generation engine determines that the corrected counterfactual action is safe. Training data is constructed by pairing safe counterfactual actions and dangerous initial actions, and the embodied multimodal large model is updated by combining reinforcement learning algorithms.

2. The method as described in claim 1, characterized in that, The action correction engine performs safety corrections on the initial dangerous action based on the risk description to generate a counterfactual action, specifically including: The action correction engine extracts keywords that characterize danger from the risk description, and fine-tunes or logically corrects the initial action that poses a danger based on the keywords to generate counterfactual actions. The fine-tuning includes correcting at least one of the motion path, motion height, and motion speed of the initial action; the logical correction includes changing the motion path of the initial action.

3. The method as described in claim 1, characterized in that, Training data is constructed by pairing safe counterfactual actions and dangerous initial actions, and the embodied multimodal large model is updated using reinforcement learning, specifically including: Preference pairs are constructed based on safe counterfactual actions and dangerous initial actions: (I_t, T, a'_t, a_t), where I_t represents the current environment image, T represents the user command, a'_t represents the safe counterfactual action, a_t represents the dangerous initial action, and a'_t corresponds to high reward and a_t corresponds to high penalty. Using the aforementioned preference pairs, the parameters of the embodied multimodal large model are updated using reinforcement learning algorithms that employ direct preference optimization or proximal policy optimization, so that it generates actions that conform to physical safety constraints.

4. The method as described in claim 1, characterized in that, The multimodal security discriminator is a semantic-level physical security discriminator; therefore, the future image and preset security warning words are input into the multimodal security discriminator for analysis, specifically including: The semantic-level physical safety discriminator receives future images and preset safety warning words, and combines them with common sense about physics to make causal inferences about the potential consequences of future images as long-term tasks; wherein, the common sense about physics includes: easy to spill, fragile, conductive, and high temperature.

5. The method according to any one of claims 1-4, characterized in that, The preset probability threshold ranges from 0.4 to 0.

8.

6. The method as described in claim 1, characterized in that, When the initial action is determined to be safe, the method further includes: The robot corresponding to the embodied multimodal large model directly executes the initial action.

7. A secure post-training system for embodied multimodal large models, characterized in that, include: The system comprises an embodied multimodal large model, an image-generative world model, a multimodal safety discriminator, a reward generation engine, an action correction engine, and a reinforcement learning optimizer; among these, The embodied multimodal large model is used to receive user instructions and acquire current environmental images, make predictions based on the user instructions and the current environmental images, and output the initial action corresponding to the user instructions. The image-generative world model is used to receive the initial action and the current environment image, make predictions, and output a future image that will occur when the initial action is performed. The multimodal security discriminator is used to receive the future image and preset security prompt words, analyze them, and output a security probability and risk description that characterizes whether the initial action is safe or not. The reward generation engine is used to receive the safety probability and make a judgment. If the safety probability is lower than a preset probability threshold, the initial action is determined to be dangerous; if the safety probability is not lower than the preset probability threshold, the initial action is determined to be safe. The action correction engine is used to make safety corrections to the initial dangerous action based on the risk description, so as to generate a counterfactual action and return it to the image-generated world model for verification. This process is repeated until the reward generation engine determines that the corrected counterfactual action is safe. The reinforcement learning optimizer is used to construct pairs of training data based on safe counterfactual actions and dangerous initial actions, and to update the embodied multimodal large model in conjunction with reinforcement learning algorithms.

8. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images