Model training method and apparatus, device, and storage medium

Abstract

The present application provides a model training method and apparatus, a device, and a storage medium. The method comprises: acquiring a training sample set, wherein training samples in the training sample set comprise a sample query request and a sample image corresponding to a sample task executed by a target device, and the sample task is associated with the sample query request; using a first machine learning model to be trained, to determine, on the basis of the sample query request and the sample image, a first execution action corresponding to the target device; using a trained second machine learning model to determine, on the basis of text information associated with the sample image as well as the sample query request, a second execution action corresponding to the target device; determining a target loss value on the basis of the first execution action, the second execution action, and a predetermined loss function; and training the first machine learning model on the basis of the target loss value. The present application can effectively improve the performance of the first machine learning model, and improves the accuracy of the first machine learning model.

Description

Model training methods, devices, equipment, and storage media

[0001] This application claims priority to Chinese Patent Application No. 202411463580.8, filed on October 18, 2024, entitled "Model Training Method, Apparatus, Device and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] The exemplary embodiments disclosed herein relate generally to the field of computers, and more particularly to model training methods, apparatus, devices, and computer-readable storage media. Background Technology

[0003] Embodied intelligence is a hot topic in current artificial intelligence research. It represents a crucial step towards achieving artificial general intelligence (AGI), where embodied agents can fully understand their environment and perform various complex tasks through real-time interaction with the physical world. This technology has broad application potential, bringing significant convenience to people's lives in areas such as autonomous driving, smart homes, healthcare, and industrial production. Improving the accuracy of this interaction is a key focus. Summary of the Invention

[0004] In a first aspect of this disclosure, a model training method is provided. The method includes: acquiring a training sample set, wherein the training samples in the training sample set include sample query requests and sample images corresponding to a target device performing a sample task, the sample task being associated with the sample query request; using a first machine learning model to be trained, based on the sample query request and the sample images, determining a first execution action corresponding to the target device; using a trained second machine learning model, based on text information associated with the sample images and the sample query request, determining a second execution action corresponding to the target device; determining a target loss value based on the first execution action, the second execution action, and a predetermined loss function; and training the first machine learning model based on the target loss value.

[0005] In a second aspect of this disclosure, an interaction method is provided. The method includes: acquiring a target query request; acquiring a target image corresponding to a target device, wherein the target image is an image acquired by the target device while performing a target task at the current time, and the target task is associated with the target query request; using a trained first machine learning model, based on the target query request and the target image, determining a first target execution action corresponding to the target device; and controlling the target device to execute the first target execution action.

[0006] In a third aspect of this disclosure, an apparatus for model training is provided. The apparatus includes: a first acquisition module configured to acquire a training sample set, wherein the training samples in the training sample set include sample query requests and sample images corresponding to a sample task performed by a target device, the sample task being associated with the sample query request; a first determination module configured to determine a first execution action corresponding to the target device based on the sample query request and the sample images using a first machine learning model to be trained; a second determination module configured to determine a second execution action corresponding to the target device based on text information associated with the sample images and the sample query request using a trained second machine learning model; a third determination module configured to determine a target loss value based on the first execution action, the second execution action, and a predetermined loss function; and a training module configured to train the first machine learning model based on the target loss value.

[0007] In a fourth aspect of this disclosure, an apparatus for interaction is provided. The apparatus includes: a second acquisition module configured to acquire a target query request; a third acquisition module configured to acquire a target image corresponding to a target device, wherein the target image is an image acquired by the target device while performing a target task at the current time, and the target task is associated with the target query request; a fourth determination module configured to determine a first target execution action corresponding to the target device based on the target query request and the target image using a trained first machine learning model; and a control module configured to control the target device to execute the first target execution action.

[0008] In a fifth aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the methods of the first or second aspect.

[0009] In a sixth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the methods of the first or second aspect.

[0010] In a seventh aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method of the first or second aspect.

[0011] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0013] Figure 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0014] Figure 2 shows a flowchart of the model training process according to some embodiments of the present disclosure;

[0015] Figure 3 shows an example system block diagram for cross-modal imitation learning according to some embodiments of the present disclosure;

[0016] Figure 4 illustrates a flowchart of an interaction process according to some embodiments of the present disclosure;

[0017] Figure 5 shows a block diagram of an apparatus for model training according to some embodiments of the present disclosure;

[0018] Figure 6 shows a block diagram of an interactive device according to some embodiments of the present disclosure; and

[0019] Figure 7 shows a block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented. Detailed Implementation

[0020] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0021] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0022] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0023] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0024] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0025] Embodied intelligence brings tremendous convenience to human life in many fields. For example, autonomous driving systems can utilize embodied intelligence to perceive road conditions in real time, make intelligent decisions, and precisely control vehicles. Another example is household robots, which can use embodied intelligence to interact with their environment, complete various household chores and entertainment tasks, and provide customized, personalized services. Furthermore, embodied intelligence can also be applied to complex operational tasks, such as robotic assembly in industrial production and surgical assistance in the medical field.

[0026] While large language models (LLMs) perform well in simulated textual worlds, they lack the ability to interpret perceptual information from other modalities (such as visual or audio), making them difficult to interact with the real world and unsuitable for direct application to real-world scenarios with multiple perceptual modalities. Furthermore, training embodied agents in a purely visual world is challenging and inefficient due to the complexity of the visual world, the low-density semantic information of visual pixels, and noise in visual signals. Although existing state-of-the-art large visual language models (VLMs) align LLMs with visual models, their training focuses only on the static alignment between image-text pairs, failing to adequately handle the dynamic alignment of the visual world. Therefore, directly applying these agents to complex embodied intelligence tasks still yields unsatisfactory performance.

[0027] In addition, most VLMs currently equipped with large language models are not the most powerful because the most powerful language models are closed-source and can only be called through the API provided by OpenAI, and cannot be directly fine-tuned based on the ChatGPT model. This, to some extent, limits the development of visual language multimodal agents.

[0028] The embodiments of this disclosure propose a model training scheme, which involves obtaining a training sample set, wherein the training samples in the training sample set include sample query requests and sample images corresponding to sample tasks performed by the target device, and the sample tasks are associated with the sample query requests; using a first machine learning model to be trained, based on the sample query requests and sample images, determining a first execution action corresponding to the target device; using a trained second machine learning model, based on text information associated with the sample images and the sample query requests, determining a second execution action corresponding to the target device; and determining a target loss value using a predetermined loss function; and training the first machine learning model based on the target loss value.

[0029] According to embodiments of this disclosure, a knowledge distillation or imitation learning method is used to train the first machine learning model to be trained based on cross-modal imitation learning between the first machine learning model and the second machine learning model. This can avoid the influence of noise interference and other factors that may occur during training in a purely visual world on the first machine learning model, and enable the first machine model to be trained to improve its performance and accuracy by learning the reasoning and reflection capabilities of the second machine learning model.

[0030] Example Environment

[0031] Figure 1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. As shown in Figure 1, the example environment 100 may include an electronic device 110. As shown in Figure 1, a first machine learning model 130-1 (the first machine learning model to be trained) with parameter values ​​before training and a first machine learning model 130-2 (the first machine learning model after training) with parameter values ​​after training may be collectively or individually referred to as the first machine learning model 130. The first machine learning model 130-1 and the second machine learning model 150 are implemented or included in the electronic device 110. The second machine learning model 150 is used to assist the first machine learning model 130-1 in learning the reasoning and reflection capabilities of the second machine learning model.

[0032] In environment 100 of Figure 1, it is desirable to train and use a machine learning model (i.e., model 130) that is configured for a variety of application environments, specifically for multimodal application environments.

[0033] As an example, electronic device 110 may contain a model training system, and electronic device 120 may contain a model application system; that is, the model training system and the model application system may exist in different electronic devices. As another example, electronic device 110 may contain both a model training system and a model application system; that is, the model training system and the model application system may exist in the same electronic device (not shown in Figure 1).

[0034] Before training, the parameter values ​​of model 130 can have initial values, or they can have pre-trained parameter values ​​obtained through a pre-training process. In some embodiments, electronic device 110 can train model 130-1 via forward and backward propagation, during which the parameter values ​​of model 130-1 can be updated and adjusted. After training is completed, model 130-2 can be obtained. At this time, the parameter values ​​of model 130-2 have been updated, and based on the updated parameter values, model 130-2 can be used for real-time interaction between the agent and the physical world during the model application phase.

[0035] During the model training phase, model 130 can be trained using a model training system based on a training sample set comprising multiple training samples 112. Specifically, the training process can be performed iteratively using a large number of training samples. After training is complete, model 130 can include knowledge about the task to be processed. During the model application phase, model 130 (at this point, model 130 has the trained parameter values) can be used to perform the corresponding task. For example, it can receive a target query request input by a user, and the target image acquired by the target device in performing the target task at the current time as input to model 130-2, and output the corresponding model output, which can be any appropriate action expected to be performed by the target device. The target device can be any appropriate device, such as an intelligent robot, smart home device, etc.

[0036] In some embodiments, the model training system may also consist of a construction component (e.g., a sample construction subsystem) for building training samples and a training component (e.g., a training subsystem) for training the model, thereby separating the sample construction and model training processes. For example, the model training system may consist of a set of devices, with one part serving as the construction component and another part serving as the training component. In some embodiments, the model training system may also utilize the same devices to build training samples and train the model based on the training samples. This disclosure is not intended to be limiting.

[0037] For ease of understanding, the following examples will all use the same equipment to build training samples and train the model based on the training samples.

[0038] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0039] Example process

[0040] Figure 2 shows a flowchart of a model training process 200 according to some embodiments of the present disclosure. Process 200 can be implemented at an electronic device 110. Process 200 is described below with reference to Figure 1.

[0041] In box 210, electronic device 110 acquires a training sample set, wherein the training samples in the training sample set include sample query requests and sample images corresponding to the sample tasks performed by the target device, and the sample tasks are associated with the sample query requests.

[0042] In some embodiments, a sample query request can be a request that represents the user's intent. It can be any appropriate type of request, such as a query request in the form of text, image, voice, or other forms.

[0043] In some embodiments, the target device can be any suitable smart device that supports performing predetermined actions. As an example, the target device can be a robot, a smart home device, etc.

[0044] In some embodiments, the sample task can be any suitable task associated with the sample query request; that is, the sample task is the task that the target device needs to complete in order to fulfill the request representing the user's intent. In some embodiments, the sample image is an image associated with the target device that is acquired by the target device during the execution of the sample task. This image can be acquired by the target device or by other devices.

[0045] As an example, the sample query request could be "Please get me an apple", the sample task could be the task of getting an apple, and the sample image could be an image collected by the intelligent robot during the process of receiving the sample query request and starting to perform the task of getting an apple.

[0046] It should be noted that the target device can perform one execution action or multiple execution actions during the execution of the sample task, depending on the difficulty of the task and the environmental information in which the task is performed, etc., which will not be elaborated here.

[0047] In box 220, electronic device 110 uses a first machine learning model to be trained to determine the first execution action corresponding to the target device based on the sample query request and the sample image.

[0048] In some embodiments, the first machine learning model can be any suitable deep learning network or deep learning model. The first machine learning model can receive multimodal input information, such as text input information, image input information, etc., and possesses universality and practicality. As an example, the first machine learning model can be an Embodied Multi-Modal Agent (EMMA).

[0049] In some embodiments, the electronic device 110 can input a sample query request and a sample image into a first machine learning model to obtain a first execution action corresponding to the target device output by the first machine learning model. The first execution action is the next action that the first machine learning model expects the target device to perform based on the sample query request and the sample image.

[0050] Figure 3 illustrates an example system block diagram of cross-modal imitation learning according to some embodiments of the present disclosure. Taking Figure 3 as an example, electronic device 110 can input image A and query request B into multimodal agent 301, which can output execution action 302 based on image A and query request B, wherein multimodal agent 301 is an example of a first machine learning model.

[0051] Taking the multimodal agent structure framework 310 shown in Figure 3 as an example, it can be built on top of the pre-trained InstructBLIP model. The multimodal agent 310 may include modules such as a Vision Transformer (ViT) encoder 311, a Query Transformer (Q-Former) module 314, a Linear Projection Layer 315, and a Large Language Model Encoder (LLM Decoder) 316.

[0052] After image A and query request B are input into multimodal agent 310, the visual Transformer encoder in multimodal agent 301 can extract visual features 312 from image A. This ViT encoder can be a frozen ViT encoder. After the Q-Former module obtains visual features 312 and features 313 corresponding to query request B, it can output a target feature, which can then be input into a frozen large language model decoder 316 after passing through a linear projection layer 315. The large language model decoder 316 can obtain the execution action 302 determined based on image A and query request B.

[0053] In some embodiments, the multimedia agent 310 can take a query request B (which may be a user-input instruction) and an image A as input, and then generate high-level text actions. Similar to other visual-language agents, the multimedia agent 310 is fine-tuned on a pre-collected embodied multimodal dataset to acquire the grammar of the ALF World Benchmark and basic game knowledge. It can be seen that the multimedia agent 310 primarily perceives environmental changes based on pixel-level images.

[0054] In box 230, the electronic device uses a trained second machine learning model to determine a second action corresponding to the target device based on text information associated with the sample image and the sample query request.

[0055] In some embodiments, the second machine learning model can be any suitable deep learning network or deep learning model. As an example, the second machine learning model can be a large language model; specifically, the second machine learning model can be the text-davinci-003 model, or other variations thereof.

[0056] The text information associated with the sample image can be other types of information with a different representation than that of the sample image. For example, it can be text information describing the image content included in the sample image. As an example, if the sample image is an image of a table with an apple on the table, the text information could be "the apple is on the table".

[0057] In some embodiments, the electronic device 110 may obtain a sample image corresponding to the target device and text information associated with the sample image based on a predetermined simulator.

[0058] Taking Figure 3 as an example, electronic device 110 can obtain image A and its corresponding text information based on simulator 305. The text information corresponding to image A can be the text information obtained after encoding image A by the state / observation descriptor in simulator 305. Specifically, electronic device 110 can extract metadata for each state from the simulator, including attributes such as observed objects, relationships between observed objects, quantity, and location. Then, we use Planning Domain Definition Language (PDDL) to describe the metadata as environmental states in text form.

[0059] In some embodiments, the electronic device 110 can input the sample query request and the text information corresponding to the sample image into the second machine learning model to obtain the second execution action corresponding to the target device output by the second machine learning model. The second execution action is the next action that the target device is expected to perform based on the sample query request and the text information corresponding to the sample image predicted by the second machine learning model.

[0060] Taking Figure 3 as an example, the electronic device 110 can input the text information corresponding to image A and the query request B into the large language model 303. The large language model 303 can output the execution action 304 based on the text information corresponding to image A and the query request B. The large language model 303 is an example of the second machine learning model.

[0061] The process of determining the second action by the second machine learning model will be explained further below.

[0062] In some embodiments, the electronic device 110 may utilize the actuator in the second machine learning model to determine a second execution action corresponding to the target device based on target short-term memory data and target long-term memory data. Target short-term memory data refers to information used for temporary storage and processing in artificial intelligence and machine learning systems. In some embodiments, the target short-term memory data may be associated with first task data corresponding to historical times prior to the current time, where the first task data is data corresponding to the target device performing a sample task. The first task data may include historical action trajectories during the execution of the sample task by the target device, which include the history of each action performed during the execution of the sample task.

[0063] In some embodiments, the target long-term memory data is experiential data that is persistently stored and accessible for a long period in an artificial intelligence and machine learning system; this experiential data can also be referred to as summary data. In some embodiments, the target long-term memory data can be determined by the commenter in the second machine learning model based on second task data corresponding to historical times prior to the current time. The second task data is data corresponding to historical tasks performed by the target device. The second task data may include historical action trajectories during the execution of the sample task by the target device, as well as historical trajectories during the execution of other tasks, etc., which will not be elaborated here.

[0064] In some embodiments, the electronic device 110 can generate first input information for the actuator based on target short-term memory data and target long-term memory data. In some embodiments, the first input information may include a first prompt, which indicates the role information corresponding to the actuator and how to determine the execution action based on this role information and the input information. As an example, the first prompt could be, "You are an execution agent. Your current task is to determine the robot's next action based on the input information. Specifically, you can determine the next action based on this input information and the actions the robot has performed in the past." In some embodiments, the electronic device 110 can obtain the second execution action corresponding to the target device output by the actuator based on the first input information.

[0065] In some embodiments, the electronic device 110 can generate second input information for the commentator based on second task data corresponding to historical times prior to the current time. In some embodiments, the second input information may include a second prompt, which indicates the commentator's corresponding role information and how to summarize experience based on this role information, etc. As an example, the first prompt could be "You are a commentator, and your current task is to summarize experience based on the input information." The electronic device 110 can obtain the target long-term memory data output by the commentator based on the second input information.

[0066] Taking the structural framework 320 of the large language model shown in Figure 3 as an example, it may include the executor 321 of the large language model and the commenter 324 of the large language model.

[0067] After the text information corresponding to image A and query request B are input into the large language model 303, the executor 321 (also called the actor) of the large language model can use query request B and text information as short-term memory data 323. The commentator of the large language model can summarize based on query request B, text information, and short-term memory data to obtain long-term memory data 322. The executor 321 of the large language model can determine the action 304 based on the long-term memory data 322 and short-term memory data 323. It can be seen that the large language model 303 can provide expert actions for the multimodal agent 301, or act as a commentator (LLM Critic) to analyze the historical trajectory of the multimodal agent and update the actor's (LLM Actor) prompts to generate improved actions in subsequent training. It can be seen that the large language model 303 mainly perceives changes in the environment through the text environment description output by the state / observation descriptor.

[0068] In the TextWorld virtual environment engine, an LLM-based agent generates a second action sequentially based on the textual environmental state and task instructions as input. In this way, we can use an LLM-driven agent to interact with TextWorld, obtain a series of expert actions, and thus acquire cross-modal imitation learning data.

[0069] In box 240, electronic device 110 determines a target loss value based on a first execution action, a second execution action, and a predetermined loss function.

[0070] In some embodiments, the predetermined loss function can be any suitable function, such as the cross-entropy loss function, the mean squared error function, etc., which will not be elaborated here.

[0071] In box 250, electronic device 110 trains the first machine learning model based on the target loss value.

[0072] The electronic device can update the parameters of the first machine learning model using gradient descent based on the target loss value. The electronic device can obtain the trained first machine learning model after determining that predetermined training conditions have been met. Training conditions could include reaching a predetermined threshold of training iterations or the target loss value reaching its minimum, etc., which will not be elaborated upon here.

[0073] Since the target device may need to perform multiple actions to complete the sample task, and after each action is completed, before determining the next action, it needs to acquire the corresponding image and then determine the next action based on the image, the visual world is dynamic. This disclosure can dynamically align the first machine learning model with the visual world and extract the ability of a large language model in the parallel text world to train a first machine learning model with reflective capabilities in the visual world. This not only improves the first machine learning model's environmental understanding ability in the dynamic visual world, but also performs dynamic visual world fine-tuning learning on offline multimodal datasets, and enables the first machine learning model to learn the correct decision-making strategy in the multimodal environment from the behavior of experts, thereby improving its performance in visual understanding, task execution, and environmental adaptation.

[0074] According to embodiments of this disclosure, a knowledge distillation or imitation learning method is used to train the first machine learning model to be trained based on cross-modal imitation learning between the first machine learning model and the second machine learning model. This can avoid the influence of noise interference and other factors that may occur during training in a purely visual world on the first machine learning model, and enable the first machine model to be trained to improve its performance and accuracy by learning the reasoning and reflection capabilities of the second machine learning model.

[0075] Figure 4 shows a flowchart of an interaction process 400 according to some embodiments of the present disclosure. Taking the example that process 400 is implemented at electronic device 120, process 400 is described below with reference to Figure 1.

[0076] In box 410, electronic device 120 obtains the target query request.

[0077] In some embodiments, the target query request is a query request input by the user indicating the user's intent. In some embodiments, the electronic device 120 may determine the first query request as a target query request in response to receiving a first query request input by the user at the current time; that is, if the target device has not performed any action for the target task, the target query request may be a query request input by the user. In other embodiments, the electronic device 120 may determine the target query request based on the second query request and the second target execution action in response to receiving a second query request input by the user at a predetermined time before the current time, and after the target device has performed a second target execution action. That is, if the target device has performed an action for the target task, in order to improve the accuracy of determining the second target execution action and improve the interactive experience, the electronic device 120 may concatenate the query request input by the user and the information corresponding to the actions performed by the target device for the target task to obtain a more comprehensive query request that is helpful for the first machine learning model to identify.

[0078] In box 420, electronic device 120 acquires the target image corresponding to the target device, wherein the target image is the image collected by the target device when performing the target task at the current time, and the target task is associated with the target query request.

[0079] The target image is the image captured by the target device when performing the target task at the current time. The target task is associated with the target query request. For example, if the target query request is "Please play a song for me", then the target task can be a music playback task.

[0080] In box 430, electronic device 120 uses a trained first machine learning model to determine the first target action corresponding to the target device based on the target query request and the target image.

[0081] The first target action is the next action that the target device is expected to perform based on the target query request and the target image, determined by the first trained machine learning model.

[0082] In frame 440, electronic device 120 controls target device to perform the first target execution action.

[0083] For example, if the first target's action is "Please walk to the living room", then the electronic device can control the target device to walk from its current location to the living room.

[0084] According to the embodiments of this disclosure, the accuracy of the interaction can be effectively improved, and the user's interactive experience can be enhanced.

[0085] Example devices and equipment

[0086] Embodiments of this disclosure also provide corresponding apparatus for implementing the methods or processes described above. Figure 5 shows a schematic structural block diagram of an apparatus 500 for model training according to certain embodiments of this disclosure. The apparatus 500 may be implemented as or included in the electronic device 110 discussed above. The various modules / components in the apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0087] As shown in Figure 5, the device 500 includes a first acquisition module 510, configured to acquire a training sample set, wherein the training samples in the training sample set include sample query requests and sample images corresponding to sample tasks performed by the target device, and the sample tasks are associated with the sample query requests; a first determination module 520, configured to use a first machine learning model to be trained, based on the sample query requests and sample images, to determine a first execution action corresponding to the target device; a second determination module 530, configured to use a trained second machine learning model, based on text information associated with the sample images and the sample query requests, to determine a second execution action corresponding to the target device; a third determination module 540, configured to determine a target loss value based on the first execution action, the second execution action, and a predetermined loss function; and a training module 550, configured to train the first machine learning model based on the difference between the first execution action and the second execution action.

[0088] In some embodiments, the second determining module 530 is further configured to: utilize the executor in the second machine learning model to determine a second execution action corresponding to the target device based on the target short-term memory data and the target long-term memory data, wherein the target short-term memory data is associated with first task data corresponding to historical time before the current time, the first task data is data corresponding to the target device executing a sample task, and wherein the target long-term memory data is determined by the commenter in the second machine learning model based on the second task data corresponding to historical time before the current time, the second task data is data corresponding to the historical tasks executed by the target device.

[0089] In some embodiments, the second determining module 530 is further configured to: generate first input information for the actuator based on the target short-term memory data and the target long-term memory data; and obtain a second execution action corresponding to the target device output by the actuator based on the first input information.

[0090] In some embodiments, the apparatus 500 further includes a first generation module configured to: generate second input information for the commentator based on second task data corresponding to historical times prior to the current time; and a second acquisition module configured to: acquire target long-term memory data output by the commentator based on the second input information.

[0091] In some embodiments, the apparatus 500 further includes a first acquisition module configured to: acquire a sample image corresponding to the target device and text information associated with the sample image based on a predetermined simulator.

[0092] Figure 6 shows a schematic structural block diagram of an interactive device 600 according to certain embodiments of the present disclosure. The device 600 may be implemented as or included in the electronic device 120 discussed above. The various modules / components in the device 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0093] As shown in Figure 6, the device 600 includes a second acquisition module 610 configured to acquire a target query request; a third acquisition module 620 configured to acquire a target image corresponding to a target device, wherein the target image is an image collected by the target device when performing a target task at the current time, and the target task is associated with the target query request; a fourth determination module 630 configured to use a trained first machine learning model to determine a first target execution action corresponding to the target device based on the target query request and the target image; and a control module 640 configured to control the target device to execute the first target execution action.

[0094] In some embodiments, the second acquisition module 610 is further configured to: determine the first query request as a target query request in response to receiving a first query request input by a user at the current time.

[0095] In some embodiments, the second acquisition module 610 is further configured to: in response to receiving a second query request input by a user at a predetermined time before the current time, and after the target device has completed a second target execution action, determine a target query request based on the second query request and the second target execution action.

[0096] The units included in device 500 and / or device 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 500 and / or device 600 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0097] Figure 7 shows a block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 700 shown in Figure 7 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. The electronic device 700 shown in Figure 7 can be used to implement the electronic device 110 or the electronic device 120 shown in Figure 1.

[0098] As shown in Figure 7, the electronic device 700 is in the form of a general-purpose electronic device. Components of the electronic device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage devices 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processing unit 710 may be a physical or virtual processor and is capable of performing various processes according to programs stored in the memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 700.

[0099] Electronic device 700 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 can be a removable or non-removable medium and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within electronic device 700.

[0100] Electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 7, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. Memory 720 may include computer program product 725 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0101] The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0102] Input device 750 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 700 can also communicate with one or more external devices (not shown) via communication unit 740 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 700, or with any device that enables electronic device 700 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0103] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0104] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0105] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0106] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0107] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0108] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for training a model, comprising: Obtain a training sample set, wherein the training samples in the training sample set include sample query requests and sample images corresponding to the sample tasks performed by the target device, and the sample tasks are associated with the sample query requests; Using the first machine learning model to be trained, based on the sample query request and the sample image, determine the first execution action corresponding to the target device; Using a trained second machine learning model, based on text information associated with the sample image and the sample query request, a second execution action corresponding to the target device is determined; Based on the first execution action, the second execution action, and a predetermined loss function, determine the target loss value; and The first machine learning model is trained based on the target loss value.

2. The method according to claim 1, wherein determining the second execution action corresponding to the target device based on text information associated with the sample image and the sample query request using a trained second machine learning model comprises: Using the actuator in the second machine learning model, based on the target short-term memory data and the target long-term memory data, the second execution action corresponding to the target device is determined. The target short-term memory data is associated with first task data corresponding to historical times prior to the current time. This first task data is the data corresponding to the target device performing the sample task. The target long-term memory data is determined by the commenter in the second machine learning model based on the second task data corresponding to the historical time prior to the current time, and the second task data is the data corresponding to the historical tasks executed by the target device.

3. The method according to claim 2, wherein determining the second execution action corresponding to the target device based on the target short-term memory data and the target long-term memory data using the executor in the second machine learning model includes: Based on the target short-term memory data and the target long-term memory data, the first input information of the actuator is generated; as well as Obtain the second execution action corresponding to the target device output by the actuator based on the first input information.

4. The method according to claim 2, further comprising: The second input information for the commenter is generated based on the second task data corresponding to historical times prior to the current time. as well as Obtain the target long-term memory data output by the commenter based on the second input information.

5. The method according to claim 1, further comprising: Based on a predetermined simulator, a sample image corresponding to the target device and text information associated with the sample image are obtained.

6. An interaction method, comprising: Obtain the target query request; Obtain the target image corresponding to the target device, wherein the target image is an image collected by the target device when performing the target task at the current time, and the target task is associated with the target query request; Using a trained first machine learning model, based on the target query request and the target image, a first target execution action corresponding to the target device is determined, wherein the trained first machine learning model is trained based on the model training method of any one of claims 1-5; as well as Control the target device to perform the first target execution action.

7. The method according to claim 6, wherein obtaining the target query request includes: In response to receiving a first query request input by the user at the current time, the first query request is determined to be the target query request.

8. The method according to claim 6, wherein obtaining the target query request includes: In response to receiving a second query request input by a user at a predetermined time prior to the current time, and after the target device has completed a second target execution action, the target query request is determined based on the second query request and the second target execution action.

9. An apparatus for model training, comprising: The first acquisition module is configured to acquire a training sample set, wherein the training samples in the training sample set include a sample query request and a sample image corresponding to the target device performing a sample task, and the sample task is associated with the sample query request. The first determining module is configured to use a first machine learning model to be trained to determine a first execution action corresponding to the target device based on the sample query request and the sample image. The second determining module is configured to use a trained second machine learning model to determine a second execution action corresponding to the target device based on text information associated with the sample image and the sample query request. The third determining module is configured to determine the target loss value based on the first execution action, the second execution action, and a predetermined loss function; The training module is configured to train the first machine learning model based on the target loss value.

10. A device for interaction, comprising: The second acquisition module is configured to acquire the target query request; The third acquisition module is configured to acquire a target image corresponding to the target device, wherein the target image is an image collected by the target device when performing a target task at the current time, and the target task is associated with the target query request; The fourth determining module is configured to use a trained first machine learning model to determine a first target execution action corresponding to the target device based on the target query request and the target image, wherein the trained first machine learning model is trained based on the model training method of any one of claims 1-5; as well as The control module is configured to control the target device to perform the first target execution action.

11. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform the method according to any one of claims 1 to 5 or the method according to any one of claims 6 to 8.

12. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1 to 5 or the method according to any one of claims 6 to 8.

13. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 5 or the method according to any one of claims 6 to 8.

Images