Robot action generation method and device, electronic equipment and medium

Abstract

The embodiment of the invention provides a robot action generation method and device, electronic equipment and a medium, belongs to the technical field of artificial intelligence, and is applied to financial scenes and medical scenes. The method comprises the following steps: performing action task analysis on an action control instruction through a robot action generation model to obtain a target action task, and distributing a basic task step length for the target action task; adjusting the basic task step length according to the target image and the ontology sensing data to obtain an adjusted task step length; performing weighted fusion on the basic task step length and the adjustment task step length to obtain a fused task step length; and performing action generation based on the fusion task step length to obtain a target action. According to the embodiment of the invention, operation such as action task analysis, basic task step size distribution, task step size adjustment and action generation is carried out on the control data of the target robot, and differentiated task step size control can be carried out for different task requirements, so that the action control effect of the robot is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and is applied to financial and medical scenarios, and in particular to a robot motion generation method and device, electronic equipment, and medium. Background Art

[0002] Traditional robot motion generation methods typically use a diffusion policy generated by a diffusion model to generate motion sequences based on a fixed sampling step and motion templates. For example, in a financial application scenario, a robot in a bank lobby receives a control instruction to deliver a document to a customer. Based on this control instruction, a diffusion policy is used to capture the robot's continuous actions of picking up the document, moving to the customer, and delivering the document, using a fixed sampling step (e.g., 100 steps). This fixed sampling step approach to generating robot motion results in poor motion control. Therefore, improving robot motion control has become a pressing issue. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a robot motion generation method and device, electronic equipment and medium, aiming to improve the robot's motion control effect.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application provides a robot motion generation method, the method comprising:

[0005] Obtaining motion control instructions for controlling a target robot, and obtaining proprioception data and a target image of the target robot according to the motion control instructions;

[0006] Obtain a pre-trained robot motion generation model; wherein the robot motion generation model includes a task parsing network, a step length adjustment network, and a motion generation network; the step length adjustment network includes a task step length adjustment subnetwork and a state step length adjustment subnetwork;

[0007] Performing action task analysis on the action control instruction using the task analysis network to obtain a target action task;

[0008] Assigning a basic task step size to the target action task using the task step size adjustment subnetwork;

[0009] Adjusting the basic task step length using the state step length adjustment subnetwork, the target image, and the proprioceptive perception data to obtain an adjusted task step length;

[0010] Performing weighted fusion of the basic task step length and the adjustment task step length to obtain a fusion task step length;

[0011] Action generation is performed based on the action generation network and the fusion task step to obtain a target action.

[0012] In some embodiments, adjusting the basic task step length using the state step length adjustment subnetwork, the target image, and the proprioceptive perception data to obtain the adjusted task step length includes:

[0013] Performing visual encoding on the target image using the state step adjustment subnetwork to obtain visual features;

[0014] Performing feature extraction on the ontology perception data to obtain ontology perception data features;

[0015] Performing motion state recognition on the target robot based on the visual features and the proprioception data features to obtain a target motion state;

[0016] The basic task step length is adjusted according to the target motion state to obtain the adjusted task step length.

[0017] In some embodiments, the target motion state includes an interactive motion state and a non-interactive motion state;

[0018] The step length adjustment of the basic task step length according to the target motion state to obtain the adjusted task step length includes:

[0019] If the target motion state is an interactive motion state, incrementally adjusting the basic task step length to obtain the adjusted task step length;

[0020] If the target motion state is a non-interactive motion state, the basic task step size is decremented to obtain the adjusted task step size.

[0021] In some embodiments, performing action generation based on the action generation network and the fusion task step to obtain a target action includes:

[0022] Determining the number of diffusion steps based on the action generation network and the fusion task step size;

[0023] Get initial noise data;

[0024] gradually diffusing the initial noise data into the action generation network based on the diffusion step number to obtain a noise action sequence;

[0025] The noise action sequence is gradually denoised based on the diffusion steps to obtain the target action.

[0026] In some embodiments, before determining the number of diffusion steps based on the action generation network and the fusion task step length, the method further includes:

[0027] Get the initial window size and window weight of the preset smoothing window filter;

[0028] Adjusting the initial window size according to the fusion task step to obtain an adjusted window size;

[0029] Performing window weight adjustment on the window weight according to the fusion task step to obtain an adjusted window weight;

[0030] A historical task step length is obtained, and the fusion task step length is smoothed based on the historical task step length, the adjustment window size, and the adjustment window weight to obtain a smoothed fusion task step length.

[0031] In some embodiments, performing action task parsing on the action control instruction using the task parsing network to obtain a target action task includes:

[0032] Extracting semantic features of the action control instructions using the task parsing network to obtain control instruction features;

[0033] Performing initial action task analysis on the control instruction feature to obtain an initial action task;

[0034] Decomposing the initial action task to obtain action subtasks;

[0035] Identifying task accuracy requirements for the action subtask to obtain task accuracy requirements;

[0036] The target action task is determined based on the action subtasks and the task accuracy requirement.

[0037] In some embodiments, performing weighted fusion of the basic task step length and the adjustment task step length to obtain the fused task step length includes:

[0038] Obtaining a basic step weight of the basic task step, and obtaining an adjustment step weight of the adjustment task step;

[0039] The basic task step length and the adjustment task step length are weightedly fused based on the basic step length weight and the adjustment step length weight to obtain the fused task step length.

[0040] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a robot motion generation device, the device comprising:

[0041] A motion control data acquisition module, configured to acquire motion control instructions for controlling a target robot, and to acquire proprioceptive perception data and a target image of the target robot according to the motion control instructions;

[0042] An action generation model acquisition module is used to acquire a pre-trained robot action generation model; wherein the robot action generation model includes a task parsing network, a step length adjustment network, and an action generation network; the step length adjustment network includes a task step length adjustment subnetwork and a state step length adjustment subnetwork;

[0043] An action task parsing module, configured to perform action task parsing on the action control instruction using the task parsing network to obtain a target action task;

[0044] A basic step length matching module is used to allocate a basic task step length to the target action task using the task step length adjustment subnetwork;

[0045] a step length adjustment module, configured to adjust the basic task step length using the state step length adjustment subnetwork, the target image, and the proprioceptive perception data to obtain an adjusted task step length;

[0046] A step fusion module is used to perform weighted fusion of the basic task step and the adjustment task step to obtain a fusion task step;

[0047] The action generation module is used to generate actions based on the action generation network and the fusion task step to obtain the target action.

[0048] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0049] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method of the above-mentioned first aspect.

[0050] The robot motion generation method and device, electronic device and medium proposed in this application help the robot to fully perceive task requirements and environmental conditions by obtaining motion control instructions, body perception data and target images for controlling the target robot, providing a rich data basis for subsequent motion generation; furthermore, the control data of the target robot are subjected to motion task analysis, basic task step allocation, task step adjustment and motion generation operations through the robot motion generation model, and differentiated task step control can be performed according to different task requirements, thereby allocating more task steps to high-precision tasks, ensuring the accuracy of generated motions, reducing task steps for low-precision tasks, and improving the efficiency of generated motions while ensuring that the accuracy of generated motions remains unchanged, thereby improving the motion control effect of the robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flowchart of a robot action generation method provided by an embodiment of the present application;

[0052] Figure 2 yes Figure 1 Flowchart of step S103 in FIG.

[0053] Figure 3 yes Figure 1 Flowchart of step S105 in FIG.

[0054] Figure 4 yes Figure 3 Flowchart of step S304 in FIG.

[0055] Figure 5 yes Figure 1 Flowchart of step S106 in FIG.

[0056] Figure 6 is another flow chart of the robot action generation method provided by an embodiment of the present application;

[0057] Figure 7 yes Figure 1 Flowchart of step S107 in FIG.

[0058] Figure 8 Schematic diagram of the structure of the robot motion generation device provided in an embodiment of the present application;

[0059] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0061] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0063] First, let’s analyze some of the terms used in this application:

[0064] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0065] The embodiments of the present application provide a robot motion generation method and device, electronic equipment, and medium, aiming to improve the motion control effect of the robot.

[0066] The robot motion generation method and device, electronic device and medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the robot motion generation method in the embodiments of the present application is described.

[0067] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0068] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0069] The robot motion generation method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The robot motion generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the robot motion generation method, etc., but is not limited to the above forms.

[0070] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0071] Figure 1 This is an optional flowchart of the robot action generation method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0072] Step S101: obtaining a motion control instruction for controlling a target robot, and obtaining proprioception data and a target image of the target robot according to the motion control instruction.

[0073] Step S102, obtaining a pre-trained robot motion generation model; wherein the robot motion generation model includes a task parsing network, a step length adjustment network and a motion generation network; the step length adjustment network includes a task step length adjustment subnetwork and a state step length adjustment subnetwork.

[0074] Step S103: Analyze the motion control instruction into a motion task using a task analysis network to obtain a target motion task.

[0075] Step S104: Use the task step adjustment subnetwork to assign basic task step sizes to the target action task.

[0076] Step S105 , adjusting the basic task step size using the state step size adjustment subnetwork, the target image, and the proprioceptive perception data to obtain an adjusted task step size.

[0077] Step S106: Perform weighted fusion on the basic task step length and the adjustment task step length to obtain a fused task step length.

[0078] Step S107: generate an action based on the action generation network and the fusion task step to obtain the target action.

[0079] Steps S101 to S107 shown in the embodiment of the present application help the robot to fully perceive task requirements and environmental conditions by acquiring motion control instructions, body perception data and target images for controlling the target robot, and provide a rich data basis for subsequent motion generation; furthermore, the control data of the target robot are subjected to motion task analysis, basic task step allocation, task step adjustment and motion generation operations through the robot motion generation model, and differentiated task step control can be performed according to different task requirements, thereby allocating more task steps to high-precision tasks, ensuring the accuracy of generated motions, reducing task steps for low-precision tasks, and improving the efficiency of generated motions while ensuring that the accuracy of generated motions remains unchanged, thereby improving the motion control effect of the robot.

[0080] In step S101 of some embodiments, specifically, the action control instruction refers to a control instruction issued by a user, which is used to guide the robot to complete a task. The action control instruction can be a text instruction in a natural language form.

[0081] For example, in a financial application scenario, the action control instruction may be “Please deliver the account opening documents to the customer”; in a medical application scenario, the action control instruction may be “Please deliver the medicine to the ward”.

[0082] Specifically, proprioception data refers to the robot's own motion state data, including but not limited to the robot's joint position, joint speed, gripper state, etc.

[0083] Specifically, the target image is an environmental image obtained by the robot's visual sensor and is used to perceive the position and state of the target object.

[0084] For example, in a financial application scenario, if the robot receives the action control instruction "Please deliver the account opening documents to the customer", the robot can obtain the position and speed of its own joints through joint sensors, and obtain the position and status image of the documents through visual sensors; in a medical application scenario, if the robot receives the action control instruction "Please deliver the medicines to the ward", the robot can obtain the position and speed of its own joints through joint sensors, and obtain the position and status image of the medicines through visual sensors.

[0085] In this embodiment, by obtaining motion control instructions and obtaining the proprioception data and target image of the target robot according to the motion control instructions, the robot can fully perceive the task requirements and environmental conditions, ensuring the accuracy and consistency of subsequent action generation.

[0086] In step S102 of some embodiments, the robot motion generation model is specifically a neural network model composed of a task parsing network, a step length adjustment network, and a motion generation network. The step length adjustment network is further subdivided into a task step length adjustment subnetwork and a state step length adjustment subnetwork.

[0087] Specifically, the task parsing network can be a large language model (LLM), which is used to parse the action task information of the action control instructions; the task step adjustment subnetwork can be a task adapter (Task-BasedAdaptor), which is used to allocate the basic task step according to the task requirement accuracy; the state step adjustment subnetwork can be a state adapter (State-Based Adaptor), which is used to further adjust the basic task step based on the target image and proprioception data; the action generation network can be a diffusion model-based policy network (Diffusion Policy) which is used to generate the final robot action according to the fused task step and adjustment step.

[0088] See also Figure 2 In some embodiments, step S103 includes but is not limited to steps S201 to S205:

[0089] Step S201 : Using a task parsing network to extract semantic features from action control instructions to obtain control instruction features.

[0090] Step S202: performing initial action task analysis on the control instruction feature to obtain the initial action task.

[0091] Step S203: Decompose the initial action task into action subtasks.

[0092] Step S204: identify the task accuracy requirement of the action subtask to obtain the task accuracy requirement.

[0093] Step S205: determining the target action task based on the action subtasks and task accuracy requirements.

[0094] In step S201 of some embodiments, specifically, the control instruction feature refers to an action control semantic vector extracted from the action control instruction.

[0095] Specifically, the action control instruction can be encoded into a control instruction token through a text encoding layer (such as a Bert model).

[0096] For example, in a financial application scenario, when a robot receives the instruction to "deliver documents to a customer", the task parsing network can extract key semantic feature vectors such as "customer", "delivery", and "document"; in a medical application scenario, for the instruction to "deliver medicines to the ward", semantic feature vectors such as "medicines" and "ward" are extracted.

[0097] In step S202 of some embodiments, specifically, the initial action task may be obtained by performing initial action task analysis on the control instruction feature.

[0098] For example, in financial applications, control instructions are characterized by key semantic vectors such as "customer," "delivery," and "file." Through initial action task parsing, these features are converted into specific action tasks, such as "move to the file location and grab the file." In medical applications, control instructions are characterized by semantic vectors such as "drug" and "ward." The parsed initial action task can be "move to the medicine cabinet and grab the medicine."

[0099] In this embodiment, the control instruction features are subjected to initial motion task analysis to obtain the initial motion task, thereby ensuring that the robot can extract specific executable motion tasks from abstract instructions.

[0100] In step S203 of some embodiments, task decomposition specifically analyzes the semantics and context information of the initial action task to decompose the initial action task into multiple specific subtasks.

[0101] For example, in a financial application scenario, when a robot receives the initial action task of "delivering documents to a client," it splits the task into multiple subtasks: moving to the document location, grabbing the document, moving to the client's location, and delivering the document. In a medical application scenario, for the initial action task of "delivering medicine to the ward," the task is split into multiple subtasks: moving to the medicine cabinet, grabbing the medicine, moving to the ward, and delivering the medicine.

[0102] In step S204 of some embodiments, specifically, the task required accuracy refers to the accuracy that the robot needs to achieve when performing the action subtask.

[0103] Specifically, the task requirement accuracy can be determined based on whether the robot interacts with the environment. If the robot needs to interact with the environment, it means that the task requirement accuracy is high. If the robot only moves itself, it means that the task requirement accuracy is low.

[0104] For example, in financial applications, the "grab a file" action subtask requires the robot to interact with the file in the environment, indicating that the task requires high precision. In the "move to a file location" action subtask, only the robot itself moves, indicating that the task requires low precision. In medical applications, the "grab a drug" action subtask requires the robot to interact with the drug in the environment, and the drug may have irregular shapes and sizes, indicating that the task requires high precision.

[0105] In this embodiment, by identifying the task precision requirements of the action subtask, it is helpful for the robot to adjust the precision of the action when performing the task, so as to improve the success rate of task execution.

[0106] In step S205 of some embodiments, specifically, the target motion task refers to a specific, decomposed motion sequence that the robot needs to complete when executing the motion control instruction.

[0107] For example, in a financial application scenario, the target action task can be "grab files with high-precision task requirements and move them quickly to the customer's location with low-precision task requirements"; in a medical application scenario, the target action task can be "grab medicines with high-precision task requirements and move them safely to the ward with low-precision task requirements."

[0108] Through steps S201 to S205, the motion control instructions are deeply analyzed through the task analysis network, and the robot can accurately understand the task requirements and decompose them into specific subtasks. The robot's motion control accuracy can be improved through precise task analysis and task decomposition mechanisms.

[0109] In step S104 of some embodiments, specifically, the basic task step length refers to the initial sampling step length that the robot needs to take when performing the action task.

[0110] Specifically, the action task with higher task precision requirement requires a larger task step size; the action task with lower task precision requirement requires a smaller task step size, so as to subsequently improve the efficiency of generating actions.

[0111] For example, in a financial application scenario, for the target action task of "grabbing files," the task step adjustment subnetwork assigns a basic task step size (e.g., 80 steps) based on the step size of historical grasping actions to ensure action precision. In a medical application scenario, for the target action task of "grabbing medicines," the task step adjustment subnetwork also assigns a basic task step size (e.g., 80 steps) based on the step size of historical grasping actions.

[0112] In this embodiment, by using the task step adjustment subnetwork to assign basic task steps to the target action task, a clear initial sampling step can be provided for the robot, which helps the robot to generate corresponding actions according to the requirements of the task.

[0113] See also Figure 3 In some embodiments, step S105 includes but is not limited to steps S301 to S304:

[0114] Step S301: Use the state step adjustment subnetwork to visually encode the target image to obtain visual features.

[0115] Step S302: extract features from the ontology perception data to obtain features of the ontology perception data.

[0116] Step S303 , identifying the motion state of the target robot based on the visual features and proprioception data features to obtain the target motion state.

[0117] Step S304 , adjusting the basic task step length according to the target motion state to obtain an adjusted task step length.

[0118] In step S301 of some embodiments, specifically, the visual feature refers to a vector representation of key information such as the position, shape, size, and state of the object extracted from the target image.

[0119] Specifically, the target image can be visually encoded through the image encoding layer to convert the target image into a set of numerical vectors, namely visual tokens, which capture features such as the position, shape, size, and state of the object in the target image.

[0120] For example, in a financial application scenario, if a target image shows a document 0.5 meters in front of the robot, the image coding layer will extract this location information as a visual feature. In a medical application scenario, if a target image shows a drug 0.6 meters in front of the robot, the image coding layer will extract this location information as a visual feature.

[0121] In step S302 of some embodiments, specifically, the ontology perception data feature refers to a feature extracted from the ontology perception data.

[0122] For example, in financial applications, when a robot performs tasks in a bank lobby, it can use joint sensors to extract joint angle and joint velocity features, gripper sensors to extract gripper angle features, torque sensors to extract torque and motor current features, and velocity sensors to extract motion speed, acceleration, and angular velocity features. In medical applications, when a robot delivers medicine in a hospital, it can also use joint sensors to extract joint angle and joint velocity features, gripper sensors to extract gripper angle features, and velocity sensors to extract motion speed, acceleration, and angular velocity features.

[0123] In step S303 of some embodiments, specifically, the target motion state is a dynamic behavior mode of the robot determined by integrating visual features and proprioception data features. The target motion state includes but is not limited to the robot's movement state, speed, and posture.

[0124] For example, in a financial application scenario, when a bank service robot performs the task of "delivering documents to customers," its visual features may include the location of the filing cabinet and obstacles on the path, and its proprioceptive data features may include the robot's current joint angles and movement speed. This information is combined to determine whether the robot is approaching the target object or whether it needs to adjust its movement speed to avoid obstacles. In a medical application scenario, when a nursing robot performs the task of "delivering medicine to the ward," its visual features may include the location of the medicine cabinet and the path to the ward, and its proprioceptive data features may include the robot's current balance state and load condition. This information is combined to determine whether the robot is in the stage of grasping the medicine or whether it needs to adjust the gripper angle or posture to grasp the medicine.

[0125] In this embodiment, the motion state of the target robot is identified based on visual features and proprioception data features. The robot can dynamically adjust its behavior according to real-time visual and proprioception information to ensure the accuracy of subsequent action generation.

[0126] See also Figure 4In some embodiments, the target motion state includes an interactive motion state and a non-interactive motion state. Step S304 includes, but is not limited to, steps S401 to S402:

[0127] Step S401: If the target motion state is an interactive motion state, the basic task step length is incrementally adjusted to obtain an adjusted task step length.

[0128] Step S402: If the target motion state is a non-interactive motion state, the basic task step length is decremented to obtain an adjusted task step length.

[0129] In step S401 of some embodiments, specifically, the target motion state may include an interactive motion state and a non-interactive motion state.

[0130] Specifically, the interactive motion state refers to a task state in which the robot needs to have direct contact or interaction with people or other objects.

[0131] For example, in financial application scenarios, the motion state of a bank service robot grabbing text is an interactive motion state; in medical application scenarios, the motion state of a nursing robot grabbing medicine is also an interactive motion state.

[0132] Specifically, the non-interactive motion state refers to the state in which the robot completes the task independently without direct contact or interaction with people or other objects.

[0133] For example, in a financial application scenario, the robot's motion state when autonomously patrolling in a bank lobby is a non-interactive motion state; in a medical application scenario, the robot's motion state when moving to the medicine is also a non-interactive motion state.

[0134] Specifically, when the target motion state is an interactive motion state, the robot needs to perform incremental adjustments, that is, increase the basic task step length to ensure that the task can be completed more accurately and stably during the interaction process.

[0135] For example, in financial applications, when a banking service robot is grasping documents, the target motion state is interactive, and the task requires high precision. Therefore, the robot needs to adjust the basic task step length from 80 to 100 steps to ensure accurate document delivery during customer interactions. In medical applications, when a nursing robot is grasping medications, the target motion state is also interactive, and the task requires high precision. However, due to the irregular shape of the medications, more precise gripping control is required, so the basic task step length needs to be adjusted from 80 to 120 steps.

[0136] In step S402 of some embodiments, specifically, when the target motion state is a non-interactive motion state, the robot needs to reduce the task step length to improve the efficiency of task execution.

[0137] For example, in a financial application scenario, when a bank service robot autonomously patrols a bank lobby, the target motion state is a non-interactive motion state. The basic task step length can be adjusted from 80 to 60 steps to improve patrol efficiency and reduce unnecessary energy consumption. In a medical application scenario, when a nursing robot moves to a medication dispenser, the target motion state is a non-interactive motion state. Since the movement distance is short, the basic task step length can be adjusted from 80 to 50 steps.

[0138] Through steps S401 to S402, by dynamically adjusting the basic task step size according to the target motion state, the robot can achieve a balance between precision and efficiency in different task scenarios. Specifically, in the interactive motion state, the automatic incremental adjustment of the step size helps to subsequently generate actions with higher precision and stability, ensuring the accuracy of the robot's actions when interacting with people or objects; in the non-interactive motion state, the automatic decremental adjustment of the step size can improve the robot's motion efficiency with lower precision requirements, reduce the robot's energy consumption and shorten the task execution time, realizing a mechanism for flexibly adjusting the step size according to task requirements, ensuring the high-precision execution of interactive tasks, and improving the efficiency of non-interactive tasks.

[0139] Through steps S301 to S304, by combining visual features and proprioception data features, the robot's motion state can be comprehensively reflected, and accurate identification of the robot's motion state can be achieved. The sampling step size is increased for interactive target motion states to ensure motion accuracy, and the sampling step size can be reduced for non-interactive target motion states to improve motion execution efficiency, realizing adaptive sampling step size control, and solving the problem of generating robot motions by using a fixed sampling step size, resulting in poor robot motion control effect.

[0140] See also Figure 5 In some embodiments, step S106 includes but is not limited to steps S501 to S502:

[0141] Step S501 : obtaining a basic step weight of a basic task step, and obtaining an adjustment step weight of an adjustment task step.

[0142] Step S502 : performing step-weighted fusion of the basic task step and the adjustment task step based on the basic step weight and the adjustment step weight to obtain a fused task step.

[0143] In step S501 of some embodiments, specifically, the basic step weight is used to adjust the proportion of the basic task step in the robot motion generation process.

[0144] Specifically, the adjustment step weight is used to adjust the proportion of the adjustment task step in the robot motion generation process.

[0145] In step S502 of some embodiments, specifically, the fusion task step size refers to the sampling step size obtained by weighted adjustment of the basic task step size and the adjustment task step size.

[0146] For example, if the basic task step is 80 steps, the adjustment task step is 120 steps, the basic step weight is 0.4, and the adjustment step weight is 0.6, then the fusion task step = (80×0.4)+(120×0.6)=104 steps.

[0147] In this embodiment, the fusion task step size takes into account both the requirements of the basic task and the requirements of real-time adjustment, so that the robot can better complete the task in a complex environment.

[0148] Through steps S501 to S502, by weighted fusion of the basic task step and the adjustment task step, the robot can generate a more optimized fusion task step, so that the robot can dynamically adjust the sampling step according to the required accuracy of the task and the complexity of the environment, while ensuring the safety and reliability of the task while ensuring the efficiency of task completion.

[0149] See also Figure 6 In some embodiments, the robot motion generation method further includes but is not limited to steps S601 to S604:

[0150] Step S601: Obtain an initial window size and window weight of a preset smoothing window filter.

[0151] Step S602 : adjusting the initial window size according to the fusion task step size to obtain an adjusted window size.

[0152] Step S603: Adjust the window weight according to the fusion task step size to obtain an adjusted window weight.

[0153] Step S604 , obtaining the historical task step length, and smoothing the fusion task step length based on the historical task step length, the adjustment window size, and the adjustment window weight to obtain a smoothed fusion task step length.

[0154] In step S601 of some embodiments, specifically, the smoothing window filter is an algorithmic tool for processing a sampling step sequence, which smoothes data fluctuations by applying a sliding window to the sampling step and calculating the weighted average of the sampling steps within the window to reduce the impact of noise.

[0155] For example, in financial application scenarios, when a bank service robot is performing a task, the task step size may fluctuate dramatically due to environmental changes or task switching. At this time, the smoothing window filter can smooth the step size changes and ensure the stability of the action.

[0156] Specifically, the initial window size and window weight are two key parameters of the smoothing window filter. The initial window size determines the number of data points contained in the window, while the window weight determines the contribution of each data point in calculating the weighted average.

[0157] For example, in financial application scenarios, when a banking service robot performs a document delivery task, the initial window size may be set to include the data points of the last five task steps, and the window weight may decrease in chronological order to give more importance to the most recent data; in medical application scenarios, when a nursing robot performs a drug delivery task, the initial window size may be set to include the data points of the last ten task steps, and the window weight may be adjusted according to the urgency and importance of the task to ensure that the smoothed data can better reflect the needs of the current task.

[0158] In step S602 of some embodiments, specifically, adjusting the window size is used to adjust the number of data points included in the window.

[0159] For example, in financial applications, if a bank service robot needs to move quickly to a customer's location, the fusion task step size may be short (e.g., 50 steps). In this case, the initial window size is adjusted from 5 to 3 to reduce smoothing and preserve more motion details. In medical applications, if a nursing robot needs to precisely grasp medications, the fusion task step size may be long (e.g., 150 steps). In this case, the initial window size is adjusted from 15 to 20 to increase smoothing and ensure motion stability.

[0160] In step S603 of some embodiments, specifically, dynamically adjusting the window weight according to the characteristics of the fusion task step size can better reflect the importance of different data points.

[0161] For example, in financial applications, if the fusion task step size is small, the weights of data points that are close in time can be reduced to avoid over-reliance on a single data point. In medical applications, if the fusion task step size is large, it may be necessary to place greater emphasis on the stability of the current task. The weights of all data points within the window can be increased to smooth out motion jitter caused by task switching.

[0162] In step S604 of some embodiments, the smoothed fusion task step size refers to the task step size required by the current task determined by combining historical task step size data and the adjusted window parameters.

[0163] For example, if the current fusion task step size is 80 steps, the historical task step size is [95, 100, 105, 110, 115], the adjusted window size is 10, and the window weight is Gaussian distribution, by calculating the weighted average, the smoothed fusion task step size is 91.

[0164] Through steps S601 to S604, the fusion task step is smoothed by combining the historical task step, the adjusted window size and the window weight, which can reduce the drastic changes in the fusion task step when the task is switched or the interaction state changes, thereby improving the stability and consistency of the robot motion control, and significantly improving the robot motion control effect.

[0165] See also Figure 7 In some embodiments, step S107 includes but is not limited to steps S701 to S704:

[0166] Step S701: Determine the number of diffusion steps based on the action generation network and the fusion task step size.

[0167] Step S702: Acquire initial noise data.

[0168] Step S703: gradually diffuse the initial noise data into the action generation network based on the number of diffusion steps to obtain a noise action sequence.

[0169] Step S704 , performing step-by-step denoising on the noise action sequence based on the diffusion steps to obtain the target action.

[0170] In step S701 of some embodiments, specifically, the diffusion step number is a key parameter in the action generation network, and the diffusion step number is used to determine the number of iterations in the process of gradually generating the target action sequence from the initial noise data.

[0171] Specifically, the number of diffusion steps is determined based on the fusion task step size. The number of diffusion steps combines the accuracy requirements and real-time status of the task to ensure the efficiency and quality of action generation.

[0172] In step S702 of some embodiments, specifically, the initial noise data refers to a noise vector randomly generated from a Gaussian distribution during the action generation process, which is used as the initial input of the action generation network to provide randomness for the generation of the action sequence.

[0173] In the specific implementation process, the acquisition of initial noise data is completed through a random number generator.

[0174] For example, a Gaussian noise vector of dimension D may be generated by a random number generator, where each element zi follows a standard normal distribution N(0,1).

[0175] In this embodiment, by obtaining initial noise data, a random starting point is provided for the action generation network, so that the robot can generate diverse action sequences under different task requirements.

[0176] In step S703 of some embodiments, specifically, the noise action sequence refers to an action sequence containing random noise generated by gradually adding initial noise data to the action generation network during the action generation process.

[0177] Specifically, noise can be added to each fusion task step through the forward diffusion of the diffusion model, and a target action sequence containing noise can be gradually constructed.

[0178] For example, in a financial application scenario, if a bank service robot needs to move quickly to a customer's location and deliver documents, the fusion task step length is 80 steps. Based on this step length, the action generation network determines the number of diffusion steps to be 40 steps. Starting from the initial noise data, through 40 steps of forward diffusion, the noise data is gradually added to the action generation network to generate an action sequence containing noise.

[0179] In step S704 of some embodiments, specifically, the target action refers to an action sequence generated after gradually removing random noise data.

[0180] Specifically, the random noise data in the noise action sequence can be gradually removed through the reverse diffusion of the diffusion model to restore the target action sequence.

[0181] For example, in financial applications, starting with a noisy action sequence, denoising it through 40 diffusion steps gradually reduces the noise, ultimately generating a smooth and coherent action for quickly grabbing a file. In medical applications, starting with a noisy action sequence, denoising it through 60 diffusion steps can generate the target action for accurately grabbing medicine.

[0182] Through steps S701 to S704, the initial noise data is gradually diffused into the action generation network through the diffusion steps, and the noise action sequence is gradually denoised. With more diffusion steps, the robot can generate more refined action sequences, or with fewer diffusion steps, the efficiency of action sequence generation can be improved. This enables the robot to dynamically adjust the generated actions under different task requirements, ensuring the efficiency and accuracy of the generated actions.

[0183] The embodiments of the present application help the robot to fully perceive task requirements and environmental conditions by acquiring motion control instructions, body perception data and target images for controlling the target robot, providing a rich data basis for subsequent motion generation; furthermore, the control data of the target robot are subjected to motion task analysis, basic task step allocation, task step adjustment and motion generation operations through the robot motion generation model, and differentiated task step control can be performed according to different task requirements, thereby allocating more task steps to high-precision tasks, ensuring the accuracy of generated motions, and reducing task steps for low-precision tasks. While ensuring that the accuracy of generated motions remains unchanged, the efficiency of generated motions is improved, thereby improving the motion control effect of the robot.

[0184] See also Figure 8 The present application also provides a robot motion generation device that can implement the above-mentioned robot motion generation method. The device includes:

[0185] A motion control data acquisition module is used to obtain motion control instructions for controlling the target robot, and obtain proprioception data and target images of the target robot according to the motion control instructions;

[0186] The action generation model acquisition module is used to obtain a pre-trained robot action generation model; wherein the robot action generation model includes a task parsing network, a step length adjustment network and an action generation network; the step length adjustment network includes a task step length adjustment subnetwork and a state step length adjustment subnetwork;

[0187] The action task parsing module is used to perform action task parsing on the action control instructions using the task parsing network to obtain the target action task;

[0188] The basic step matching module is used to assign basic task steps to the target action task using the task step adjustment subnetwork;

[0189] The step length adjustment module is used to adjust the basic task step length using the state step length adjustment subnetwork, the target image and the proprioception data to obtain the adjusted task step length;

[0190] The step-size fusion module is used to perform weighted fusion of the basic task step-size and the adjustment task step-size to obtain the fused task step-size;

[0191] The action generation module is used to generate actions based on the action generation network and the fusion task step to obtain the target action.

[0192] The specific implementation of the robot motion generation device is basically the same as the specific embodiment of the above-mentioned robot motion generation method, and will not be repeated here.

[0193] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-mentioned robot motion generation method. The electronic device can be any smart terminal including a tablet computer, an in-vehicle computer, or the like.

[0194] See also Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0195] The processor 901 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0196] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the processing system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called by the processor 901 to execute the robot motion generation method of the embodiments of this application.

[0197] Input / output interface 903, used to implement information input and output;

[0198] Communication interface 904, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0199] Bus 905 , which transmits information between various components of the device (e.g., processor 901 , memory 902 , input / output interface 903 , and communication interface 904 );

[0200] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0201] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned robot action generation method is implemented.

[0202] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0203] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0204] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0205] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0206] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0207] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0208] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0209] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0210] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0211] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0212] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0213] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A robot motion generation method, characterized in that: The method comprises: Obtaining motion control instructions for controlling a target robot, and obtaining proprioception data and a target image of the target robot according to the motion control instructions; Obtain a pre-trained robot motion generation model; wherein the robot motion generation model includes a task parsing network, a step length adjustment network, and a motion generation network; the step length adjustment network includes a task step length adjustment subnetwork and a state step length adjustment subnetwork; Performing action task analysis on the action control instruction using the task analysis network to obtain a target action task; Assigning a basic task step size to the target action task using the task step size adjustment subnetwork; Adjusting the basic task step length using the state step length adjustment subnetwork, the target image, and the proprioceptive perception data to obtain an adjusted task step length; Performing weighted fusion of the basic task step length and the adjustment task step length to obtain a fusion task step length; Action generation is performed based on the action generation network and the fusion task step to obtain a target action.

2. The method according to claim 1, characterized in that The step of adjusting the basic task step by using the state step adjustment subnetwork, the target image, and the proprioceptive perception data to obtain the adjusted task step includes: Performing visual encoding on the target image using the state step adjustment subnetwork to obtain visual features; Performing feature extraction on the ontology perception data to obtain ontology perception data features; Performing motion state recognition on the target robot based on the visual features and the proprioception data features to obtain a target motion state; The basic task step length is adjusted according to the target motion state to obtain the adjusted task step length.

3. The method according to claim 2, characterized in that The target motion state includes an interactive motion state and a non-interactive motion state; The step length adjustment of the basic task step length according to the target motion state to obtain the adjusted task step length includes: If the target motion state is an interactive motion state, incrementally adjusting the basic task step length to obtain the adjusted task step length; If the target motion state is a non-interactive motion state, the basic task step size is decremented to obtain the adjusted task step size.

4. The method according to claim 1, wherein The step of generating an action based on the action generation network and the fusion task step to obtain a target action includes: Determining the number of diffusion steps based on the action generation network and the fusion task step size; Get initial noise data; gradually diffusing the initial noise data into the action generation network based on the diffusion step number to obtain a noise action sequence; The noise action sequence is gradually denoised based on the diffusion steps to obtain the target action.

5. The method according to claim 4, characterized in that Before determining the number of diffusion steps based on the action generation network and the fusion task step length, the method further includes: Get the initial window size and window weight of the preset smoothing window filter; Adjusting the initial window size according to the fusion task step to obtain an adjusted window size; Performing window weight adjustment on the window weight according to the fusion task step to obtain an adjusted window weight; A historical task step length is obtained, and the fusion task step length is smoothed based on the historical task step length, the adjustment window size, and the adjustment window weight to obtain a smoothed fusion task step length.

6. The method according to claim 1, wherein The step of performing action task parsing on the action control instruction using the task parsing network to obtain a target action task includes: Extracting semantic features of the action control instructions using the task parsing network to obtain control instruction features; Performing initial action task analysis on the control instruction feature to obtain an initial action task; Decomposing the initial action task to obtain action subtasks; Identifying task accuracy requirements for the action subtask to obtain task accuracy requirements; The target action task is determined based on the action subtasks and the task accuracy requirement.

7. The method according to any one of claims 1 to 6, characterized in that The step of weightedly fusing the basic task step length and the adjustment task step length to obtain the fused task step length includes: Obtaining a basic step weight of the basic task step, and obtaining an adjustment step weight of the adjustment task step; The basic task step length and the adjustment task step length are weightedly fused based on the basic step length weight and the adjustment step length weight to obtain the fused task step length.

8. A robot motion generation device, characterized in that: The device comprises: A motion control data acquisition module is used to acquire motion control instructions for controlling a target robot, and to acquire proprioception data and a target image of the target robot according to the motion control instructions; An action generation model acquisition module is used to acquire a pre-trained robot action generation model; wherein the robot action generation model includes a task parsing network, a step length adjustment network, and an action generation network; the step length adjustment network includes a task step length adjustment subnetwork and a state step length adjustment subnetwork; An action task parsing module, configured to perform action task parsing on the action control instruction using the task parsing network to obtain a target action task; A basic step length matching module is used to allocate a basic task step length to the target action task using the task step length adjustment subnetwork; a step length adjustment module, configured to adjust the basic task step length using the state step length adjustment subnetwork, the target image, and the proprioceptive perception data to obtain an adjusted task step length; A step fusion module is used to perform weighted fusion of the basic task step and the adjustment task step to obtain a fusion task step; The action generation module is used to generate actions based on the action generation network and the fusion task step to obtain the target action.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the robot motion generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the robot motion generation method according to any one of claims 1 to 7 is implemented.

Images