Diffusion strategy model optimization method in combination with state memory mechanism and intelligent agent with body

Abstract

The invention provides a diffusion strategy model optimization method combined with a state memory mechanism and an intelligent agent with a body, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring state information and image information of a target object at the current moment; generating a global condition vector based on the state information and the image information; inputting the global condition vector into a diffusion network model to obtain a predicted action sequence formed by action data of a plurality of time steps; and performing state prediction by using a state estimator based on the global condition vector and the action data of the first time step in the plurality of time steps to obtain a state prediction result at the next moment. By implementing the method and the device, the generated control instruction can better fit the action and state of the target object, more accurate control is realized, and the success rate of action execution is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a diffusion strategy model optimization method and an embodied intelligent agent combined with a state memory mechanism. Background Art

[0002] In recent years, diffusion policy (DP) models based on imitation learning have garnered widespread attention in embodied intelligence tasks. These models, combining a two-frame RGB image encoder (e.g., the 224x224 pixel Vision Transformer, ViT-224) with a diffusion network, can predict continuous action sequences across multiple future frames, demonstrating robustness in simple grasping and carrying tasks.

[0003] However, related technologies still suffer from the following flaws: They lack long-term state memory. Existing DP models rely solely on very short-term visual information, failing to capture states such as "completed," "failed," and "removed" during the task's progress. Sequential dependencies are also lost. In tasks requiring strict step-by-step completion, if visual information cannot distinguish the current stage, the model cannot make correct decisions. These flaws significantly reduce the success rate in complex, multi-step scenarios (particularly those with subtle visual differences). Summary of the Invention

[0004] In view of this, the present invention provides a diffusion strategy model optimization method and an embodied intelligent agent combined with a state memory mechanism.

[0005] In a first aspect, an embodiment of the present invention proposes a diffusion strategy model optimization method combined with a state memory mechanism, including: obtaining state information and image information of a target object at the current moment; generating a global condition vector based on the state information and image information; inputting the global condition vector into a diffusion network model to obtain a predicted action sequence consisting of action data of multiple time steps; based on the global condition vector and the action data of the first time step among the multiple time steps, using a state estimator to perform state prediction to obtain a state prediction result at the next moment.

[0006] In the second aspect, an embodiment of the present invention proposes an embodied intelligent body, including: a robot body and an actuator, which executes the diffusion strategy model optimization method combined with the state memory mechanism as described in any implementation method in the first aspect, so that the robot body completes the task.

[0007] The diffusion strategy model optimization method and embodied agent, combined with a state memory mechanism, provided by embodiments of the present invention, incorporates state and image information into the constructed global condition vector. This information can be used to characterize the state and visual features of the target object, and these state and visual features are then applied to the generation of action sequences within the diffusion network model. Furthermore, in conjunction with a state estimator, the implicit state at the next moment is predicted based on the current state features, visual features, and action sequence. This allows the generated action instructions to better align with the target object's motion and state, achieving more accurate control and improving the success rate of action execution.

[0008] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 is an exemplary system architecture in which the present invention may be applied; Figure 2 A flowchart of a diffusion strategy model optimization method combined with a state memory mechanism provided by an embodiment of the present invention; Figure 3 A flowchart of another method for optimizing a diffusion strategy model in combination with a state memory mechanism provided by an embodiment of the present invention; Figure 4 A flowchart of another method for optimizing a diffusion strategy model in combination with a state memory mechanism provided by an embodiment of the present invention; Figure 5 A flowchart of another method for optimizing a diffusion strategy model in combination with a state memory mechanism provided by an embodiment of the present invention; Figure 6 A flowchart of another method for optimizing a diffusion strategy model in combination with a state memory mechanism provided by an embodiment of the present invention; Figure 7A and Figure 7B A flow chart of a method for optimizing a diffusion strategy model in combination with a state memory mechanism in an application scenario provided by an embodiment of the present invention; Figure 7CA flowchart of a diffusion strategy model optimization method combined with a state memory mechanism in an application scenario provided by another embodiment of the present invention; Figure 8 A structural block diagram of a diffusion strategy model optimization device combined with a state memory mechanism provided by an embodiment of the present invention; Figure 9 A schematic structural diagram of an electronic device suitable for executing a diffusion strategy model optimization method combined with a state memory mechanism, provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0011] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0012] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0013] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; internal connections between two components; wireless connections or wired connections. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0014] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0015] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of the diffusion strategy model optimization method combined with a state memory mechanism and an embodied intelligent agent of the present invention can be applied.

[0016] like Figure 1As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0017] Users can use terminal devices 101, 102, 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 and server 105 may be installed with various applications for enabling information communication between them, such as instant messaging applications.

[0018] Terminal devices 101, 102, 103 and server 105 can be either hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules, or as a single software or software module, and are not specifically limited here. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and are not specifically limited here.

[0019] Server 105 can provide various services through various built-in applications. The data or information required to provide these services can be obtained from terminal devices 101, 102, and 103 via network 104, or can be pre-stored locally on server 105 in various ways. Therefore, when server 105 detects that such data is already stored locally, it can choose to directly obtain such data from the local storage. In this case, exemplary system architecture 100 may also not include terminal devices 101, 102, 103 and network 104.

[0020] Because data or information processing may require a significant amount of computing resources and significant computing power, the diffusion strategy model optimization methods incorporating a state memory mechanism provided in the subsequent embodiments of the present invention are generally executed by a server 105 possessing significant computing power and resources. Accordingly, the diffusion strategy model optimization apparatus incorporating a state memory mechanism is also generally located within the server 105. However, it should also be noted that, when terminal devices 101, 102, and 103 also possess sufficient computing power and resources, the terminal devices 101, 102, and 103 may also utilize the relevant applications installed thereon to complete the various calculations previously assigned to the server 105, thereby outputting the same results as the server 105. In particular, in the presence of multiple terminal devices with varying computing power, if the relevant application determines that the terminal device in question possesses significant computing power and a significant amount of remaining computing resources, the terminal device may be allowed to perform the aforementioned calculations, thereby appropriately alleviating the computing pressure on the server 105. Accordingly, the diffusion strategy model optimization apparatus incorporating a state memory mechanism may also be located within the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also not include the server 105 and the network 104 .

[0021] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0022] Please refer to Figure 2 , Figure 2 This is a flow chart of a diffusion strategy model optimization method combined with a state memory mechanism provided by an embodiment of the present invention, wherein process 200 includes the following steps: Step 201: Acquire the current state information and image information of the target object.

[0023] This step is intended to be performed by the execution subject of the diffusion strategy model optimization method combined with the state memory mechanism (e.g. Figure 1 The server 105 shown in FIG. 105 obtains the current state information and image information of the target object. The target object may be a device or apparatus (e.g., a robot or robotic arm) that can be controlled by an execution subject to perform a corresponding action or operation. The target object's state information may include states of the target object or its components, such as action angle, action type, size, and position information. The image information may include an image of the target object captured by an image capture device (e.g., a camera).

[0024] For example, in an application scenario where the target object is a robot, the end of the robot's arm is provided with an end effector (such as a flexible gripper, a mechanical gripper, an adsorption gripper, etc.), and the gripper's claws are provided with visual / physical markers (Tags) and inertial measurement units (IMUs). The tag can be a QR code or a barcode, used to mark the position of the two grippers. Based on this position, the position information of the robot's arm and the width and size information of the two grippers can be obtained; the IMU can be set at the end of the robot's arm to detect information such as the swing angle of the robot's arm to obtain the movement angle, posture or trajectory of the robot's arm. The image information can be obtained by using a GoPro camera to capture an image of the robot's arm. The robot's status information includes width information, position information and angle information in this scenario, which can be used as a data source for continuous quantity parameters.

[0025] For application scenarios where the target object is a robotic arm, the end of the robotic arm is provided with an end effector (such as a flexible gripper, a mechanical gripper, an adsorption gripper, etc.), and the gripper is provided with a visual / physical marker (Tag) and an inertial measurement unit (IMU). The tag can be a QR code or a barcode, which is used to mark the position of the two grippers. Based on this position, the position information of the robotic arm and the width and size information of the two grippers can be obtained; the IMU can be set at the end of the robotic arm to detect information such as the swing angle of the robotic arm to obtain the movement angle, posture or trajectory of the robotic arm. The image information can be obtained by using a GoPro camera to capture an image of the robotic arm. The state information of the robotic arm includes width information, position information and angle information in this scenario, which serves as the data source for continuous quantity parameters.

[0026] Step 202: Generate a global condition vector based on the state information and the image information.

[0027] This step aims to generate a global condition vector based on the acquired state information and image information by the execution subject. The global condition vector includes vectors corresponding to the state information and image information, which are used to represent the state characteristics and image characteristics of the target object at the current moment.

[0028] Step 203: Input the global condition vector into the diffusion network model to obtain a predicted action sequence consisting of action data of multiple time steps.

[0029] This step aims to enable the aforementioned execution subject to output a predicted action sequence consisting of multiple time-step motion data using a diffusion network model based on a global condition vector. This diffusion network model is a robot action generation strategy (Diffusion Policy, DP) based on a diffusion model. By modeling visual motion strategies as a conditional denoising diffusion process, this network model enables the generation and control of complex action sequences. In this embodiment, the global condition vector is used as input, and the diffusion network model can output a predicted action sequence consisting of multiple time-step motion data.

[0030] Step 204: Based on the global condition vector and the action data of the first time step among the multiple time steps, a state estimator is used to perform state prediction to obtain a state prediction result at the next moment.

[0031] This step involves the execution subject using a state estimator to predict the state based on the global condition vector and the motion data from the first of multiple time steps, thereby obtaining a predicted state result for the next moment. The state estimator is primarily used to infer internal, non-directly measurable state variables (such as position, velocity, and posture) from inputs (such as sensor data) and outputs (such as observations). In this embodiment, the state estimator is used to predict the state at the next moment, obtaining a predicted state result.

[0032] Furthermore, the obtained state prediction result at the next moment can also be used as the current state at the next moment to participate in generating the global condition vector and participate in the subsequent processing process.

[0033] The diffusion strategy model optimization method, incorporating a state memory mechanism, provided by embodiments of the present invention, incorporates state and image information into the constructed global condition vector. This information can be used to characterize the state and visual features of the target object, and these state and visual features are then applied to the generation of action sequences within the diffusion network model. Furthermore, the method, combined with a state estimator, predicts the next implicit state based on the current state features, visual features, and action sequence. This allows the generated action instructions to better align with the target object's motion and state, achieving more accurate control and improving the success rate of action execution.

[0034] Please refer to Figure 3 , Figure 3 A flowchart of a diffusion strategy model optimization method combined with a state memory mechanism provided in an embodiment of the present disclosure, namely, for Figure 2 Step 201 in the process 200 shown provides a specific implementation method. The other steps in the process 200 are not adjusted. The specific implementation method provided in this embodiment is replaced by step 201 to obtain a new complete embodiment. The process 300 includes the following steps: Step 301: Acquire continuous quantity parameters and discrete quantity parameters of the target object, wherein the continuous quantity parameters are used to characterize width information, position information and angle information of the target object, and the discrete quantity parameters are used to characterize state information or task stage information of the target object.

[0035] This step aims to enable the execution entity to obtain continuous and discrete parameters of the target object. The continuous parameters represent relevant information about the target object (e.g., width, position, angle, etc.). For example, these continuous parameters can be obtained using visual / physical tags and an inertial measurement unit (IMU) placed on the target object or one or more components of the target object. For example, the "angle" in the relevant information may include initial angle information and / or current angle information. The initial angle information represents the initial posture of the end effector at the start of the task. The current angle information represents the target angle during subsequent task execution. The "position" in the relevant information may refer to the three-dimensional spatial position of the end effector at the current time step, which may correspond to the x, y, and z coordinates in a Cartesian coordinate system. In robotic motion planning, the model uses this position information to determine whether the robotic arm has reached the target position, such as the "grasp point" or "placement point." In state prediction or reinforcement learning, position information can be used as part of the input features to predict the next action, such as the movement of the robotic arm or the opening and closing of the gripper.

[0036] The discrete quantity parameter is used to represent the state or task stage of the target object (for example, information representing states or stages such as "grasping successful," "object placed in box," "wiping completed," or "button pressed"). In this embodiment, the discrete quantity parameter can be, for example, a vector represented by a state machine and / or one-hot encoding.

[0037] Step 302: Concatenate the continuous quantity parameters and the discrete quantity parameters to obtain a state vector.

[0038] This step aims to obtain a state vector by performing splicing based on the continuous quantity parameters and the discrete quantity parameters by the above-mentioned execution subject.

[0039] Through the above process, in an embodiment of the present invention, the basic information and state information of the target object are perceived and acquired to generate a global condition vector, which can provide a more comprehensive and complete reference vector for subsequent action prediction and state prediction, thereby improving the accuracy and success rate of subsequent processing processes.

[0040] Step 303: Capture an image of the target object.

[0041] This step is intended to enable the execution subject to obtain an image of the target object captured by a visual sensor (such as a camera, a GoPro camera, etc.).

[0042] Step 304: Input the image into the visual encoder to obtain low-dimensional image features.

[0043] This step involves the execution entity extracting image features from the image using a visual encoder and performing dimensionality reduction to obtain low-dimensional image features. The visual encoder is used to convert high-dimensional data such as images or videos into low-dimensional semantic representations, providing a comprehensible embedding space for downstream tasks such as classification, detection, and generation. For example, if the image captured by a visual sensor has a resolution of 224×224, the visual encoder can extract image features from the image and perform dimensionality reduction to obtain, for example, 1536-dimensional image features.

[0044] This step aims to use the visual encoder to extract image features and perform dimensionality reduction processing by the above-mentioned execution entity. The obtained low-dimensional image features can be used to generate a global condition vector, which can provide a more comprehensive and complete reference vector for subsequent action prediction and state prediction, thereby improving the accuracy and success rate of subsequent processing.

[0045] Furthermore, in some optional implementations of the embodiments of the present invention, step 202, the process of generating a global condition vector based on the state information and image information, primarily includes: concatenating the state vector with low-dimensional image features to obtain the global condition vector. Alternatively, concatenating continuous quantity parameters with low-dimensional image features to obtain concatenated features, performing parameter segmentation on discrete quantity parameters to form scaling parameters and offset parameters, and performing feature linear modulation based on the concatenated features, scaling parameters, and offset parameters to obtain the global condition vector.

[0046] It should be noted that the step numbers involved in the above process description are not intended to limit the order in which the steps are implemented. The order in which the status information and image information are obtained can be adjusted according to actual needs.

[0047] In this embodiment, by combining continuous quantity parameters with discrete quantity parameters, the relevant information of the target object is characterized from different dimensions. By coordinating continuous action generation with discrete state estimation, the cumulative error can be reduced, more accurate control can be achieved, and the control success rate can be improved.

[0048] In some optional implementations of the embodiments of the present invention, step 204, the process of inputting the global condition vector and the action data of the first time step in multiple time steps into the state estimator for state prediction, mainly includes: performing a gradient separation operation on the global condition vector to cut off the gradient propagation path; performing a gradient separation operation on the extraction process of the action data of the first time step; inputting the global condition vector after gradient separation and the action data of the first time step after gradient separation into the state estimator for state prediction, wherein the global condition vector is obtained by splicing the state vector and the low-dimensional image features.

[0049] This step is intended to involve the execution subject performing a gradient detachment operation ("detach") on the global condition vector and the action data for the first of the multiple time steps before inputting them into the state estimator, thereby obtaining a gradient-detached global condition vector and gradient-detached action data. In this embodiment, performing the gradient detachment operation on the global condition vector and action data separates them from the computational graph of the diffusion network model and then inputs them into the state estimator for state prediction, thereby obtaining a state prediction result for the next moment.

[0050] In the embodiment of the present invention, through the above process, the global condition vector and action data input to the state estimator are separated from the computational graph of the diffusion network model by means of a gradient separation operation. This prevents the generated gradients from being propagated to the diffusion network model during subsequent training of the state estimator, thereby preventing the state estimator training from misleading the parameters of the diffusion network model and ensuring that the diffusion network model can focus on its own goal, namely, generating a reasonable action sequence.

[0051] In some optional implementations of the embodiments of the present invention, the output of the above-mentioned state estimator may include multiple forms, among which it may be the state prediction results of the current state vector and the next state vector. For such state vectors, it is usually represented by the two states (True / False) of the vector. Alternatively, it may be represented in the form of one-hot encoding, for example, generating a one-hot encoding vector [0, 0, 0, 1, 0, ..., 0] corresponding to the "successful grasping" state. The state estimator may include a multilayer perceptron (MLP). Further, as Figure 4 As shown, the above step 204, based on the global condition vector and the action data of the first time step in multiple time steps, uses the state estimator to perform state prediction to obtain the state prediction result at the next moment, mainly includes: Step 401: Perform nonlinear mapping through the output layer of the multilayer perceptron to obtain neuron output values.

[0052] This step is intended to perform nonlinear mapping by the above-mentioned execution subject through the output layer of the state estimator, where the output layer contains neurons matched with discrete quantity parameters.

[0053] Step 402: Generate a probability distribution vector of discrete quantity parameters based on the activation function and the neuron output value.

[0054] This step aims to apply the activation function to the output values ​​of the neurons in the output layer to generate a probability distribution vector of discrete quantities.

[0055] Step 403: Generate a one-hot encoding for representing the next state vector based on the probability distribution vector and the corresponding dimension.

[0056] This step aims to set the dimension corresponding to the neuron with the highest probability to 1 and the remaining dimensions to 0 based on the probability distribution vector, thereby forming a one-hot encoding (One-Hot) to represent the next state vector.

[0057] Please refer to Figure 5 , Figure 5 This is a flow chart of another method for optimizing a diffusion strategy model in combination with a state memory mechanism provided by an embodiment of the present disclosure, wherein process 500 includes the following steps: Step 501: Acquire the current state information and image information of the target object.

[0058] Step 502: Generate a global condition vector based on the state information and the image information.

[0059] Step 503: Input the global condition vector into the diffusion network model to obtain the predicted action sequence at the next moment.

[0060] Step 504: Based on the global condition vector and the action data of the first time step among the multiple time steps, a state estimator is used to perform state prediction to obtain a state prediction result at the next moment.

[0061] The above steps 501-504 are similar to the following Figure 2 Steps 201-204 shown are consistent. For the same content, please refer to the corresponding part of the previous embodiment and will not be repeated here.

[0062] Step 505: The state prediction result at the next moment is passed to the next diffusion step to form the state information at the current moment of the next diffusion step.

[0063] In this embodiment, the state prediction result obtained in the current diffusion step can be further used as the state information of the current moment of the next diffusion step to perform state prediction for the next diffusion step.

[0064] Step 506: The state information at the current moment of the next diffusion step is used as one of the input conditions of the next diffusion step for the diffusion network model to predict the subsequent action sequence, and the state estimator is made to predict the subsequent state to obtain the state prediction result at the subsequent moment, forming a closed-loop iteration of the state prediction.

[0065] Furthermore, the diffusion strategy model optimization method combined with the state memory mechanism may also include: controlling the target object to perform corresponding actions based on the predicted action sequence and the state prediction result.

[0066] This step is intended to enable the execution subject to control the target object to perform corresponding actions based on the predicted action sequence and state prediction results after obtaining the predicted action sequence and state prediction results at the next moment.

[0067] Through the above process, the obtained state prediction results are passed to the next time step, and the diffusion reasoning process of each step is integrated into the subsequent execution process to capture long-term dependencies, thereby realizing cross-step information transmission. This can make the obtained state prediction results more consistent with the action and state of the target object, achieve more accurate control, and improve the success rate of action execution.

[0068] In some optional implementations of the embodiments of the present invention, corresponding adjustments are also proposed for the training process of the diffusion network model and state estimator described in any of the above embodiments. Figure 4 Schematic diagram of the model training method of the embodiment of the present invention. Figure 6 As shown in Figure 2, the training process mainly includes: Step 601: Use the first proportion of sample data for labeling as a training set, use the second proportion of sample data as a validation set, and combine the initial diffusion network model and the initial state estimator to add pseudo labels to the remaining unlabeled sample data.

[0069] This step aims to select a first proportion of sample data for labeling as a training set, and use a second proportion of sample data as a validation set, input the remaining unlabeled sample data into the initial diffusion network model and the initial state estimator, and add pseudo labels to each unlabeled sample data. The initial diffusion network model is an untrained diffusion network model, and the initial state estimator is an untrained state estimator. The first proportion and the second proportion can be set according to actual needs and can be the same or different. For example, 5% of the sample data can be selected for labeling to obtain a training set; then 5% of the sample data can be selected as a validation set; and pseudo labels are added to the remaining 90% of the unlabeled sample data.

[0070] In some optional implementations of the embodiments of the present invention, the remaining unlabeled sample data are input into the initial diffusion network model and the initial state estimator for training, the state of the remaining unlabeled sample data is predicted, the uncertainty of the remaining unlabeled sample data is calculated, and the preliminary trained diffusion network model and state estimator are obtained; the current remaining unlabeled sample data are input into the preliminary trained diffusion network model and state estimator to obtain a first prediction result, wherein the current remaining unlabeled sample data includes unprocessed data in the remaining unlabeled sample data after preliminary training; and pseudo labels are added to the current remaining unlabeled sample data based on the first prediction result.

[0071] In the above process, the prediction uncertainty of samples in unlabeled data is determined by calculating the entropy value of the prediction results or the multi-model voting difference. The entropy value measures the degree of confusion of the probability distribution of discrete task states, and the multi-model voting difference measures the degree of disagreement between different models in predicting the same state.

[0072] Step 602: Based on the sample data with pseudo labels, the training set and the validation set, the initial diffusion network model and the initial state estimator are trained respectively to obtain the diffusion network model and the state estimator.

[0073] This step aims to train the initial diffusion network model and the initial state estimator based on the sample data, training set and validation set with pseudo labels, so as to obtain the trained diffusion network model and state estimator.

[0074] In some optional implementations of the embodiments of the present invention, sample data with uncertainty higher than a preset threshold is labeled to obtain true state label sample data, and added to the validation set to obtain an updated validation set; the initially trained diffusion network model and state estimator are evaluated using the updated validation set, and the pseudo labels are adjusted to obtain the diffusion network model and state estimator.

[0075] Through the above process, the model's prediction results have a certain degree of credibility, as it has undergone preliminary training and optimization through evaluation on the validation set. The above process of uncertainty sampling, manual labeling, validation set construction, model evaluation, and pseudo-label expansion is repeated continuously. In each iteration, the model is evaluated using the updated validation set, and historical pseudo-label data is screened and corrected. The quality of pseudo-labels gradually improves with each iteration.

[0076] Furthermore, fine-tuning can be performed on the full sample data for 50 epochs, with an initial learning rate of 1e-4 and parameter updates using the Adam optimizer. The diffusion network model and state estimator can be trained 50 times on the full dataset to further improve their accuracy and generalization.

[0077] To deepen understanding, the present invention also provides a specific implementation scheme in combination with a specific application scenario, see Figure 7A and Figure 7B shown.

[0078] In this embodiment, a robotic arm is used as the target object for illustration. The end of the robotic arm is equipped with an end effector (such as a flexible gripper, a mechanical gripper, or an adsorption gripper), and the gripper's claws are equipped with visual / physical tags and an inertial measurement unit (IMU). The tag, which can be a QR code or barcode, marks the position of the two grippers. Based on this position, the gripper widths of the two grippers can be determined. By detecting the image coordinates of the tag's corner points, 3D pose information (including position information and Euler angles) relative to the acquisition camera can be obtained. The initial angle can be calibrated based on the relative pose between the tag coordinate system and the acquisition camera. The IMU can be installed at the end of the robotic arm to detect information such as the arm's swing angle to determine the arm's posture or trajectory. A GoPro camera with a sampling rate of 30Hz and an image size of 224×224 can be used to capture image information from the robotic arm.

[0079] In specific implementations, step 701: Obtain state information and image information of the robotic arm. Correspondingly, this state information includes information such as gripper width, end effector position along three coordinate axes, Euler angles, and initial angles, as well as the current state vector (e.g., "grasp successfully" represented by a one-hot encoding, corresponding to the one-hot encoding vector [0, 0, 0, 1, 0, ..., 0]). The gripper width, end effector position along three coordinate axes, Euler angles, and initial angles constitute 16-dimensional continuous parameters, while the state vector is a 16-dimensional discrete parameter, thus forming a 32-dimensional state feature.

[0080] Step 702: Generate a global condition vector based on the state information and image information. A visual encoder (e.g., ViT-224) outputs 1536-dimensional low-dimensional image features based on the image information. In practical applications, this may also include a two-frame X-dimensional vector representing the current state. The image features, state features, and the two-frame X-dimensional vectors are reshaped and concatenated to generate the global condition vector.

[0081] Step 703: Input the global condition vector into the diffusion network model to obtain a predicted action sequence consisting of action data for multiple time steps. The global condition vector is input into the U-Net diffusion network to predict the action sequence for multiple time steps. In this embodiment, the action sequence can be, for example, a 7-step action sequence with 16 dimensions per step.

[0082] In specific implementation, the global condition vector can also align the feature dimensions through linear transformation to adapt to the input requirements of the U-Net diffusion network.

[0083] Step 704: Based on the global condition vector and the action data for the first time step among the multiple time steps, a state estimator is used to perform state prediction, obtaining a state prediction result for the next time step. After performing a gradient separation operation on the action sequence and the global condition vector, the state estimator inputs the state prediction result for the next time step, outputting a 32+16-dimensional state prediction result for the next time step.

[0084] Step 705: Based on the predicted action sequence and the state prediction result, the robot arm is controlled to perform the corresponding action. Based on the state prediction result, the robot arm can be controlled to perform the corresponding action at the next moment.

[0085] Furthermore, the state prediction result obtained at the next moment can also be used as the current state of the next time step to generate a global condition vector and participate in the subsequent processing process.

[0086] As a possible implementation method, the present invention also provides another specific implementation scheme, referring to Figure 7C After obtaining image information through a visual sensor such as a GoPro, ViT-224 or other types of image encoders can be used to extract features, generating low-dimensional visual features / image features such as 1536 dimensions. It should be noted that image encoders can also include other types of neural networks, such as ResNet.

[0087] The continuous quantity parameters can be concatenated with the low-dimensional image features to obtain, for example, 1568-dimensional concatenated features. As another example, the continuous quantity parameters (32 dimensions) can be mapped through a linear transformation, projected into dimensionally transformed continuous quantity parameters, and then the dimensionally transformed continuous quantity parameters can be concatenated with the low-dimensional image features to form concatenated features.

[0088] This implementation also involves parameter segmentation of discrete parameters into scaling parameters (Gamma) and offset parameters (Beta). This is combined with the concatenated features for feature-wise linear modulation (FiLM). The modulated features (e.g., unet vectors) are input to the DP model as global conditions. The concatenated vectors serve as input to the FiLM MLP. Their dimensionality determines the design of the MLP's input layer, ensuring that state and task information are mapped to Gamma / Beta, effectively linking the feature modulation to the current scenario.

[0089] For example, the modulation mechanism of FiLM satisfies the following formula: Gamma × (visual features) + Beta. The feature modulation parameter film_params in FiLM is split into two parts along the last dimension: Gamma and Beta. Gamma is used for element-wise multiplication of target features (e.g., visual features) (visual feature × gamma), while Beta is used for element-wise addition of target features (visual feature + beta).

[0090] The FiLM mechanism achieves a deep fusion of state and visual information. It dynamically adjusts the importance of visual features based on different states, increasing the weight of state information in the model input and improving the model's predictive ability for different state-action combinations. Furthermore, FiLM parameters provide interpretability of model decisions, enabling state-aware visual feature modulation while maintaining high efficiency, resulting in more accurate and contextually relevant action predictions.

[0091] In the embodiments of the present invention, the control process implemented by the diffusion strategy model optimization method combined with the state memory mechanism described in any of the above embodiments can achieve the following specific effects: In the sequential sorting experiment for building blocks of different colors, the sequential sorting experiment refers to controlling the robotic arm to sort objects according to a preset color or shape sequence. In this experiment, the original DP model had a success rate of 15%, while the success rate of the embodiment of the present invention reached 90%.

[0092] In the experiment of wiping a table with a scouring pad (low color contrast), the success rate of the original DP model is almost 0, while the success rate of the embodiment of the present invention can stably reach more than 70%.

[0093] Further references Figure 8 As an implementation of the methods shown in the above figures, the present invention provides an embodiment of a diffusion strategy model optimization device combined with a state memory mechanism. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0094] like Figure 8As shown, the diffusion strategy model optimization device 800 combined with the state memory mechanism of this embodiment may include: an information acquisition module 801, a global condition vector generation module 802, an action sequence generation module 803, and a state prediction result generation module 804. The information acquisition module 801 is configured to acquire the state information and image information of the target object at the current moment; the global condition vector generation module 802 is configured to generate a global condition vector based on the state information and image information; the action sequence generation module 803 is configured to input the global condition vector into the diffusion network model to obtain a predicted action sequence consisting of action data of multiple time steps; and the state prediction result generation module 804 is configured to perform state prediction using a state estimator based on the global condition vector and the action data of the first time step among the multiple time steps to obtain a state prediction result for the next moment.

[0095] In this embodiment, in the diffusion strategy model optimization device 800 combined with the state memory mechanism, the specific processing of the information acquisition module 801, the global condition vector generation module 802, the action sequence generation module 803 and the state prediction result generation module 804 and the technical effects thereof can be referred to respectively. Figure 2 The relevant descriptions of steps 201-204 in the corresponding embodiment are not repeated here.

[0096] This embodiment, as an apparatus embodiment corresponding to the aforementioned method embodiment, provides a diffusion strategy model optimization device incorporating a state memory mechanism. This device incorporates state and image information into the constructed global condition vector, enabling characterization of the state and visual features of the target object. These state and visual features are then applied to the generation of action sequences by the diffusion network model. Furthermore, in conjunction with a state estimator, the device predicts the next implicit state based on the current state features, visual features, and action sequence. This allows the generated action instructions to better align with the target object's motion and state, achieving more accurate control and improving the success rate of action execution.

[0097] According to an embodiment of the present invention, the present invention also provides an electronic device, which includes: at least one executor; and a memory communicatively connected to the at least one executor; wherein the memory stores instructions that can be executed by the at least one executor, and the instructions are executed by the at least one executor so that when the at least one executor executes, it can implement the diffusion strategy model optimization method combined with the state memory mechanism described in any of the above embodiments.

[0098] According to an embodiment of the present invention, the present invention also provides a readable storage medium, which stores computer instructions, which are used to enable a computer to implement the diffusion strategy model optimization method combined with the state memory mechanism described in any of the above embodiments when executed.

[0099] According to an embodiment of the present invention, the present invention further provides a computer program product, which, when executed by an executor, can implement the diffusion strategy model optimization method combined with the state memory mechanism described in any of the above embodiments.

[0100] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0101] like Figure 9 As shown, device 900 includes an executor 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. Executor 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to bus 904.

[0102] Multiple components in device 900 are connected to I / O interface 905, including: an input unit 906, such as a keyboard, mouse, etc.; a robotic arm 907; a storage unit 908, such as a magnetic disk, optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0103] Actuator 901 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of actuator 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable actuator, executor, micro-actuator, etc. Actuator 901 performs the various methods and processes described above, such as the diffusion strategy model optimization method incorporating a state memory mechanism. For example, in some embodiments, the diffusion strategy model optimization method incorporating a state memory mechanism can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed onto device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by actuator 901, one or more steps of the diffusion strategy model optimization method incorporating a state memory mechanism described above can be performed to control robotic arm 907 to perform corresponding actions. Alternatively, in other embodiments, the executor 901 may be configured in any other appropriate manner (for example, by means of firmware) to execute the diffusion strategy model optimization method combined with the state memory mechanism.

[0104] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0105] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0106] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0107] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0108] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will readily appreciate that other variations or modifications based on the above descriptions are possible. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A diffusion strategy model optimization method combined with a state memory mechanism, characterized in that: include: Obtain the current state information and image information of the target object; generating a global condition vector based on the state information and the image information; Inputting the global condition vector into a diffusion network model to obtain a predicted action sequence consisting of action data of multiple time steps; Based on the global condition vector and the action data of the first time step among the multiple time steps, a state estimator is used to perform state prediction to obtain a state prediction result at the next moment.

2. The method according to claim 1, characterized in that The obtaining of the state information and image information of the target object at the current moment includes: Acquire continuous quantity parameters and discrete quantity parameters of the target object, wherein the continuous quantity parameters are used to characterize width information, position information, and angle information of the target object, and the discrete quantity parameters are used to characterize the state or task stage of the target object; splicing the continuous quantity parameters and the discrete quantity parameters to obtain a state vector; Acquiring an image of the target object; The image is input into a visual encoder to obtain low-dimensional image features.

3. The method according to claim 2, characterized in that The formation of the global condition vector includes: Concatenate the state vector and the low-dimensional image feature to obtain the global condition vector; or The continuous quantity parameter and the low-dimensional image feature are spliced ​​to obtain a splicing feature, the discrete quantity parameter is parameter-segmented to form a scaling parameter and an offset parameter, and feature linear modulation is performed based on the splicing feature, the scaling parameter and the offset parameter.

4. The method according to claim 3, characterized in that Inputting the global condition vector and the action data of the first time step among the multiple time steps into a state estimator for state prediction includes: performing a gradient separation operation on the global condition vector to cut off a gradient propagation path; performing the gradient separation operation on the extraction process of the motion data of the first time step; The global condition vector after gradient separation and the action data of the first time step after gradient separation are input into the state estimator for state prediction. The global condition vector is obtained by concatenating the state vector and the low-dimensional image feature.

5. The method according to claim 1, wherein The state estimator includes a multilayer perceptron, and the state estimator is used to perform state prediction based on the global condition vector and the action data of the first time step in the multiple time steps to obtain a state prediction result at the next moment, including: Performing nonlinear mapping through the output layer of the multilayer perceptron to obtain neuron output values; Generate a probability distribution vector of discrete quantity parameters based on the activation function and the neuron output value; A one-hot encoding for representing a next state vector is generated based on the probability distribution vector and the corresponding dimension.

6. The method according to claim 1, wherein The method further comprises forming a next diffusion step, comprising: Passing the state prediction result at the next moment to the next diffusion step to form the state information at the current moment of the next diffusion step; The state information at the current moment of the next diffusion step is used as one of the input conditions of the next diffusion step for the diffusion network model to predict the subsequent action sequence, and the state estimator is made to predict the subsequent state to obtain the state prediction result at the subsequent moment, forming a closed-loop iteration of the state prediction.

7. The method according to any one of claims 1 to 6, characterized in that: Also includes: The first proportion of sample data is used for labeling as a training set, and the second proportion of sample data is used as a validation set. The initial diffusion network model and the initial state estimator are combined to add pseudo labels to the remaining unlabeled sample data. The initial diffusion network model and the initial state estimator are trained based on the sample data with the pseudo labels added, the training set and the validation set, respectively, to obtain the diffusion network model and the state estimator.

8. The method according to claim 7, characterized in that The method of adding pseudo labels to the remaining unlabeled sample data by combining the initial diffusion network model and the initial state estimator includes: Inputting the remaining unlabeled sample data into the initial diffusion network model and the initial state estimator for training, performing state prediction on the remaining unlabeled sample data, calculating the uncertainty of the remaining unlabeled sample data, and obtaining a preliminarily trained diffusion network model and a preliminarily trained state estimator; Inputting the currently remaining unlabeled sample data into the preliminarily trained diffusion network model and the preliminarily trained state estimator to obtain a first prediction result, wherein the currently remaining unlabeled sample data includes unprocessed data in the remaining unlabeled sample data after the preliminarily training; Add pseudo labels to the currently remaining unlabeled sample data based on the first prediction result.

9. The method according to claim 8, characterized in that The training of the initial diffusion network model and the initial state estimator based on the sample data with the pseudo labels added, the training set, and the validation set respectively includes: Labeling the sample data with uncertainty higher than a preset threshold to obtain true state label sample data, and adding the data to the validation set to obtain an updated validation set; The initially trained diffusion network model and the initially trained state estimator are evaluated using the updated validation set, and the pseudo labels are adjusted to obtain the diffusion network model and the state estimator.

10. An embodied intelligent agent, characterized in that include: Robot body; An actuator is configured to execute the diffusion strategy model optimization method combined with the state memory mechanism as described in any one of claims 1 to 9, so that the robot body completes the task.

Images