Robot interaction methods, devices, equipment and program products

Abstract

This application relates to the field of artificial intelligence, providing a robot interaction method, device, equipment, and program product. The method includes: estimating the three-dimensional pose of the interactive object in the acquired interactive video stream data to determine spatial pose information, and recognizing the interaction intent using a preset large language model to generate a response action sequence; inputting the response action sequence, the robot's current state, time step, and a randomly generated noisy action sequence into a motion prediction model to obtain the predicted action output by the motion prediction model; wherein, the motion prediction model is used to perform reverse denoising on the randomly generated noisy action sequence based on the response action sequence and the robot's current state to obtain the predicted action. The method provided by this application can meet interaction requirements while ensuring the robot's safety and task feasibility, avoiding problems such as stiff, jittery, or unnatural robot movements, resulting in a smoother final robot performance.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a robot interaction method, device, equipment, and program product. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, human-robot interaction (HRI) has become a crucial research area. Its goal is to overcome the limitations of traditional command-based interaction, enabling robots to interact with humans in a more natural and efficient way. The physical form, autonomous decision-making capabilities, and communication methods of a robot system significantly influence users' perception, trust, and willingness to participate. Therefore, developing robot systems that can integrate into human social environments and effectively collaborate with humans has become a key direction and core challenge in the current development of robotics technology.

[0003] Current human-computer interaction technologies primarily achieve this by constructing a two-way feedback loop. Users input commands or express intentions to the robot through voice, gestures, facial expressions, or touch. The robot then perceives these inputs through its onboard sensors (such as cameras, microphones, and force sensors), analyzes and understands them using built-in algorithm models, and subsequently executes corresponding tasks or generates outputs such as language or actions to respond to the human. This process forms a continuous, interdependent causal pattern: human behavior triggers the robot's response, and the robot's response, in turn, serves as new input, influencing subsequent human behavior.

[0004] However, in the aforementioned human-computer interaction technologies, robot actions are mainly generated based on information perceived by sensors combined with user-defined action combinations. However, user-defined action combinations require precise setting of start and end times, action coordination ratios, etc., which creates an operational barrier for non-professional users, easily reducing the system's ease of use and accessibility. Furthermore, it is difficult to dynamically adjust the action strategy according to the robot's own real-time status, which may lead to action execution failure or safety issues. Summary of the Invention

[0005] This application provides a robot interaction method, device, equipment, and program product to solve the technical problem that predicting actions based on perception information can easily lead to action execution failure or unsafety.

[0006] In a first aspect, embodiments of this application provide a robot interaction method, comprising: acquiring interactive video stream data; performing three-dimensional pose estimation on the interactive object in the interactive video stream data based on the interactive video stream data, determining spatial pose information, and recognizing the interaction intent in conjunction with a preset large language model to generate a response action sequence; inputting the response action sequence, the robot's current state, time step, and a randomly generated noisy action sequence into an action prediction model to obtain a predicted action output by the action prediction model; wherein, the action prediction model is trained based on an action sample sequence and corresponding robot state samples and action sequence labels, and the action prediction model is used to perform reverse denoising on the randomly generated noisy action sequence based on the response action sequence and the robot's current state to obtain the predicted action.

[0007] In one embodiment, the reaction action sequence, the robot's current state, time steps, and a randomly generated noisy action sequence are input into the action prediction model to obtain the predicted action output by the action prediction model. This includes: inputting the reaction action sequence, the robot's current state, time steps, and the randomly generated noisy action sequence into the action prediction model to embed the time steps into each noisy action of the noisy action sequence; based on each noisy action after embedding the time steps, using a self-attention mechanism to capture the contextual information of each noisy action after embedding the time steps, and combining the reaction action sequence and the robot's current state to perform reverse denoising on the corresponding noisy action after embedding the time steps, obtaining the corresponding denoised action; based on each denoised action, using a self-attention mechanism to capture the contextual information of each denoised action, and combining the reaction action sequence and the robot's current state to perform reverse denoising on the corresponding denoised action, and iteratively executing this step to progressively denoise the corresponding denoised action until a preset number of iterations is reached, obtaining the predicted action output by the action prediction model.

[0008] In one embodiment, before inputting the reaction action sequence, the robot's current state, the time step, and the randomly generated noisy action sequence into the action prediction model, the method includes: obtaining an action sample sequence and corresponding robot state samples and action sequence labels; inputting the action sample sequence and corresponding robot state samples and action sequence labels into the model to be trained, so as to positively add noise to each action in the action sequence labels to obtain a noisy action sequence, and extracting features from the input action sample sequence and robot state samples respectively, so as to combine the extracted action feature sequence and state feature sequence to perform reverse denoising on the noisy action sequence to obtain a denoised action sequence; constructing a loss function based on the denoised action sequence and the corresponding action sequence label, and obtaining an action prediction model for generating predicted actions based on the convergence of the loss function.

[0009] In one embodiment, based on interactive video stream data, a three-dimensional pose estimation is performed on the interactive object in the interactive video stream data to determine spatial pose information. This includes: performing three-dimensional pose estimation on the interactive object in the interactive video stream data to obtain the three-dimensional skeletal joint coordinates of the interactive object as pose information; determining the position and orientation of the interactive object relative to the robot based on the pose information and the robot's position coordinates to obtain spatial position information; generating scene description text based on the spatial position information and a preset text template; and fusing the pose information, spatial position information, and scene description text to obtain spatial pose information.

[0010] In one embodiment, fusion of attitude information, spatial location information, and scene description text to obtain spatial pose information includes: tokenizing the attitude information, spatial location information, and scene description text respectively to obtain corresponding attitude token sequences, spatial location token sequences, and text token sequences; and concatenating the attitude token sequences, spatial location token sequences, and text token sequences in a preset concatenation order or combining them in a preset combination order to obtain spatial pose information.

[0011] In one embodiment, tokenization of the posture information includes: normalizing the coordinates of the three-dimensional skeletal joints in the posture information, and reducing the dimensionality of the normalized posture information to obtain posture dimensionality-reduced data; encoding the posture dimensionality-reduced data to obtain a continuous posture vector with the same dimension as the posture dimensionality-reduced data; quantizing the continuous posture vector to obtain the index of the quantized posture vector, and determining the posture token; wherein, the quantized posture vector is the posture code vector that is closest to the continuous posture vector in the first codebook obtained earlier.

[0012] In one embodiment, tokenizing spatial location information includes: normalizing the spatial location information and reducing the dimensionality of the normalized spatial location information to obtain dimensionality-reduced spatial location data; encoding the dimensionality-reduced spatial location data to obtain a continuous spatial location vector with the same dimension as the dimensionality-reduced spatial location data; quantizing the continuous spatial location vector to obtain an index of the quantized spatial location vector, and determining the spatial location token; wherein the quantized spatial location vector is the spatial location code vector that is closest to the continuous spatial location vector in the previously obtained second codebook.

[0013] In one embodiment, tokenization of the scene description text includes: SA, constructing a vocabulary by using all characters of the scene description text as basic units in an initial vocabulary; SB, traversing the scene description text, counting the frequency of occurrence of all adjacent character pairs, and merging the character pair with the highest frequency to obtain a new subword; SC, updating the vocabulary with the new subword, replacing the corresponding character pair with the highest frequency in the scene description text with the new subword to obtain updated text, and iteratively executing steps SB-SC until a preset stopping condition is reached; SD, dividing the finally obtained updated text into subword tokens according to the final vocabulary to obtain a text token sequence.

[0014] In one embodiment, after obtaining the predicted action output by the action prediction model, the method includes: sending the predicted action to the robot's controller to perform an autonomous response based on the predicted action and update the robot's state.

[0015] Secondly, embodiments of this application provide a robot interaction device, comprising: a data acquisition module for acquiring interactive video stream data; a first action prediction module for estimating the three-dimensional pose of the interactive object in the interactive video stream data based on the interactive video stream data, determining spatial pose information, and recognizing the interaction intent in conjunction with a preset large language model to generate a response action sequence; and a second action prediction module for inputting the response action sequence, the robot's current state, time step, and a randomly generated noisy action sequence into an action prediction model to obtain the predicted action output by the action prediction model; wherein the action prediction model is trained based on the action sample sequence and the robot state sample and action sequence label corresponding to the action sample sequence, and the action prediction model is used to perform reverse denoising on the randomly generated noisy action sequence based on the response action sequence and the robot's current state to obtain the predicted action.

[0016] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the steps of the robot interaction method described in the first or second aspect.

[0017] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the robot interaction method described in the first or second aspect.

[0018] The robot interaction method, apparatus, device, and program products provided in this application acquire interactive video stream data to capture the dynamically changing interaction process, providing a more comprehensive context for the robot's subsequent action planning, making decision-making safer and more intelligent. They determine the spatial pose information of the interactive object through three-dimensional pose estimation, then combine this with a large language model to understand the interaction intent, ultimately generating a context-appropriate, human-like sequence of robot responses. Furthermore, by combining the robot's current state, time step, and randomly generated noisy action sequence, and using an action prediction model, the noisy action sequence is denoised based on the robot's current state and response action sequence to generate a clear, coherent, and high-quality final output. This ensures that the final predicted action is the result of optimization under global constraints, meeting interaction requirements while guaranteeing the robot's safety and task feasibility, avoiding problems such as stiff, jittery, or unnatural robot movements, resulting in a smoother, more human-like final robot performance and improved user experience. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is one of the flowcharts illustrating the robot interaction method provided in the embodiments of this application; Figure 2 This is a second flowchart illustrating the robot interaction method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the robot interaction device provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] Figure 1 This is a flowchart illustrating the robot interaction method. (Refer to...) Figure 1This application provides a robot interaction method, which may include: S11, acquire interactive video stream data; S12, Based on the interactive video stream data, perform three-dimensional pose estimation on the interactive objects in the interactive video stream data, determine the spatial pose information, and combine it with a preset large language model to identify the interactive intent and generate a sequence of reaction actions; S13, input the reaction action sequence, the robot's current state, the time step, and the randomly generated noisy action sequence into the action prediction model to obtain the predicted action output by the action prediction model; wherein, the action prediction model is trained based on the action sample sequence and the robot state sample and action sequence label corresponding to the action sample sequence. The action prediction model is used to perform reverse denoising on the randomly generated noisy action sequence based on the reaction action sequence and the robot's current state to obtain the predicted action.

[0023] It should be noted that the step numbers "S1N" in this manual do not represent the order of the robot interaction methods. The following details will explain further. Figure 2 The robot interaction method of the present invention is described.

[0024] Step S11: Obtain interactive video stream data.

[0025] It should be noted that the interactive video stream data is video data captured in real time by the robot's onboard visual sensors during the interaction process with humans. The visual sensors include RGB cameras, depth cameras, etc., and the specific type depends on the actual camera on the robot. No further restrictions are made here.

[0026] Step S12: Based on the interactive video stream data, perform three-dimensional pose estimation on the interactive objects in the interactive video stream data to determine the spatial pose information, and combine it with a preset large language model to identify the interactive intent and generate a sequence of reaction actions.

[0027] It's worth noting that the pre-defined Large Language Model (LLM) leverages its powerful contextual understanding and reasoning capabilities to analyze multimodal interaction information related to spatial pose, enabling it to autonomously understand human intentions or behaviors and determine the appropriate action the robot should take. The output of the LLM is a series of tokens representing high-level action intentions, i.e., T. action For example, if a person waves, the LLM might generate an action token representing a "wave response." During training, T... action This will be further transformed into responsive movements and postures, and then subjected to supervised training and fine-tuning.

[0028] In this embodiment, based on the interactive video stream data, a three-dimensional pose estimation is performed on the interactive object in the interactive video stream data to determine spatial pose information. This includes: estimating the three-dimensional pose of the interactive object in the interactive video stream data to obtain the three-dimensional skeletal joint coordinates of the interactive object as pose information; determining the position and orientation of the interactive object relative to the robot based on the pose information and the robot's position coordinates to obtain spatial position information; generating scene description text based on the spatial position information and a preset text template; and fusing the pose information, spatial position information, and scene description text to obtain spatial pose information.

[0029] It should be added that 3D pose estimation can utilize algorithms such as SMPLify-X (human pose estimation) and OpenPose (open pose estimation) combined with depth information. The specific algorithm chosen depends on the actual design requirements and is not further limited here. The 3D skeletal joint coordinates of the interactive object are represented as follows: ),in, Indicates the first Each key point is The three-dimensional coordinates at time [time]. This represents the total number of 3D skeletal joints of the interactive object, and... As attitude information.

[0030] In addition, the preset text template can be set according to actual design requirements. For example, it can be configured according to the parameter type of spatial location information so that after obtaining the spatial location information, its specific parameters can be filled into the preset text template. No further limitations are made here.

[0031] In addition, refer to Figure 2 The method involves fusing attitude information, spatial location information, and scene description text to obtain spatial pose information, including: tokenizing the attitude information, spatial location information, and scene description text to obtain corresponding attitude token sequences, spatial location token sequences, and text token sequences; and concatenating the attitude token sequences, spatial location token sequences, and text token sequences in a preset splicing order or combining them in a preset combination order to obtain spatial pose information.

[0032] It should be noted that the preset splicing order or preset combination order can be set according to actual design requirements, for example, it can be set to "A person standing at position".<x,y,z,y,p,r> is waving to you in pose< > No further limitations are made here. Additionally, by tokenizing the pose information, spatial location information, and scene description text separately, they are converted into a unified, discrete token sequence. This facilitates the concatenation or combination of pose token sequences, spatial location token sequences, and text token sequences, making subsequent action prediction model processing easier. This allows the robot to have a more comprehensive and profound understanding of the interaction scene, enabling it to capture non-verbal cues and achieve more intelligent interaction. The spatial pose information is represented as... ,in, These represent text token sequences, gesture token sequences, and spatial location token sequences, respectively.

[0033] Furthermore, the tokenization process for the posture information includes: normalizing the coordinates of the three-dimensional skeletal joints in the posture information, and reducing the dimensionality of the normalized posture information to obtain posture dimensionality-reduced data; encoding the posture dimensionality-reduced data to obtain a continuous posture vector with the same dimension as the posture dimensionality-reduced data; quantizing the continuous posture vector to obtain the index of the quantized posture vector, and determining the posture token; wherein, the quantized posture vector is the posture code vector that is closest to the continuous posture vector in the first codebook obtained earlier.

[0034] It should be added that by normalizing the coordinates of multiple 3D skeletal joints obtained from 3D pose estimation, the scale differences caused by different individuals, different shooting distances, and different coordinate systems are eliminated, making the 3D skeletal joint coordinates comparable. Furthermore, the dimensionality reduction process represents the dimensionality-reduced pose information as a low-dimensional, continuous vector to reduce computational complexity, remove noise, and extract the most representative features. The dimensionality reduction method can be selected according to actual design requirements, such as principal component analysis, and no further limitations are made here.

[0035] In an optional embodiment, the above-mentioned encoding and quantization of the pose dimensionality reduction data can be achieved by a vector quantization variational autoencoder (VQ-VAE). Accordingly, before determining the code vector in the preset codebook that is closest to the continuous pose based on the continuous pose vector, the process includes: training the VQ-VAE. Specifically, training a VQ-VAE includes: acquiring pose samples; normalizing the pose samples and reducing their dimensionality to obtain dimensionality-reduced pose samples; encoding the dimensionality-reduced pose samples to map them to a latent space to obtain continuous sample vectors; determining the code vector closest to the continuous sample vectors in a pre-defined codebook based on the continuous sample vectors; obtaining discrete pose tokens based on the indices corresponding to the determined code vectors, and generating discrete code vectors based on the determined code vectors and continuous sample vectors; decoding the discrete code vectors to reconstruct them back into the dimensionality-reduced pose samples to obtain decoded vectors; constructing a reconstruction loss function based on the decoded vectors and dimensionality-reduced pose samples; constructing a codebook loss function based on the discrete code vectors and the determined code vectors; and constructing a commitment loss function based on the continuous sample vectors and discrete code vectors; obtaining the total loss function based on the reconstruction loss function, codebook loss function, and commitment loss function; and converging based on the total loss function to terminate training and obtain the VQ-VAE.

[0036] In addition, the spatial location information undergoes tokenization, including: normalizing the spatial location information and reducing its dimensionality to obtain dimensionality-reduced spatial location data; encoding the dimensionality-reduced spatial location data to obtain a continuous spatial location vector with the same dimensionality as the dimensionality-reduced spatial location data; and quantizing the continuous spatial location vector to obtain the index of the quantized spatial location vector, thus determining the spatial location token. The quantized spatial location vector is the spatial location code vector that is closest to the continuous spatial location vector in the previously obtained second codebook. It should be noted that encoding and quantizing the dimensionality-reduced spatial location data can also be implemented using VQ-VAE. The training of VQ-VAE can be referred to the above description and will not be further elaborated here.

[0037] In addition, the scene description text undergoes tokenization, including: SA, using all characters in the scene description text as basic units in the initial vocabulary to construct the vocabulary; SB, traversing the scene description text, counting the frequency of all adjacent character pairs, and merging the most frequent character pair to obtain a new subword; SC, updating the vocabulary with the new subword, replacing the corresponding most frequent character pair in the scene description text with the new subword to obtain updated text, and iteratively executing steps SB-SC until a preset stopping condition is reached; SD, based on the final vocabulary, segmenting the final updated text into subword tokens to obtain a text token sequence. It should be noted that the preset stopping condition can be set according to actual design requirements, such as the number of character pair mergings or iterations, and is not further limited here.

[0038] Step S13: Input the reaction action sequence, the robot's current state, the time step, and the randomly generated noisy action sequence into the action prediction model to obtain the predicted action output by the action prediction model; wherein, the action prediction model is trained based on the action sample sequence and the robot state sample and action sequence label corresponding to the action sample sequence. The action prediction model is used to perform reverse denoising on the randomly generated noisy action sequence based on the reaction action sequence and the robot's current state to obtain the predicted action.

[0039] It should be added that the robot's current state includes the angles and velocities of each joint, the position and orientation of the end effector, and the robot's task state, etc. The specifics can be obtained based on the actual robot, and no further limitations are made here.

[0040] In this embodiment, the reaction action sequence, the robot's current state, time steps, and a randomly generated noisy action sequence are input into the action prediction model to obtain the predicted action output by the action prediction model. This includes: inputting the reaction action sequence, the robot's current state, time steps, and the randomly generated noisy action sequence into the action prediction model to embed the time steps into each noisy action of the noisy action sequence; based on each noisy action after embedding the time steps, using a self-attention mechanism to capture the contextual information of each noisy action after embedding the time steps, and combining the reaction action sequence and the robot's current state, performing reverse denoising on the corresponding noisy action after embedding the time steps to obtain the corresponding denoised action; based on each denoised action, using a self-attention mechanism to capture the contextual information of each denoised action, and combining the reaction action sequence and the robot's current state, performing reverse denoising on the corresponding denoised action, and iteratively executing this step to progressively denoise the corresponding denoised action until a preset number of iterations is reached to obtain the predicted action output by the action prediction model.

[0041] It's worth noting that the motion prediction model can employ a Diffusion Transformer (DiT). DiT combines the Transformer architecture's ability to process sequence information with the Diffusion model's generative capabilities. Diffusion models excel at generating complex, high-dimensional continuous data, resulting in smoother, more coordinated robot movements that more closely resemble human motion patterns. This significantly enhances the naturalness of human-computer interaction and user experience. By learning the complex mapping relationship between motion tokens, robot states, and the final motion trajectory through DiT, the noise required for denoising can be accurately predicted. This allows for accurate denoising of noisy actions, rapidly generating actions that match the intent and conform to physical constraints, thus improving the accuracy of motion prediction. The overall response process is efficient and real-time. Furthermore, time steps are embedded into each noisy action in the noisy motion sequence, including embedding the time steps into a preset high-order space and adding them to each noisy action through positional encoding.

[0042] Furthermore, the noise reduction action is represented as: in, Indicates a noise reduction action; This represents the noise to be denoised as predicted by the model. This represents the noisy action at time step t. The conditions include the sequence of actions that the model inputs and the robot's current state; , They represent the smoothing coefficients, ; Indicates based on The obtained standard deviation; Represents the cumulative product coefficient. ; Indicates standard Gaussian noise. It should be noted that the noise reduction process is iteratively performed according to the above formula until the desired result is obtained. As the predicted action output by the final model, the robot's current state is fully considered during the iterative denoising process to ensure that the generated action is feasible, safe, and can adapt to the dynamic changes in the robot's own state and environment.

[0043] Furthermore, based on the noisy actions after each embedding time step, a self-attention mechanism is used to capture the contextual information of the noisy actions after each embedding time step. This includes: determining the query vector, key vector, and value vector of the corresponding noisy action after each embedding time step using the self-attention mechanism; for each noisy action after an embedding time step, performing a dot product operation on the query vector of the noisy action after the embedding time step and the key vector of the noisy action after the embedding time step, respectively, to determine the corresponding attention score, and normalizing all the obtained attention scores to obtain the attention weight of the corresponding noisy action after the embedding time step; and performing a weighted summation of the attention weight of the noisy action after the embedding time step and the value vectors of the noisy actions after the embedding time step to obtain the contextual information of the noisy action after the embedding time step.

[0044] Furthermore, the normalization process can select the corresponding function according to the actual design requirements. For example, the normalization exponent Softmax function can be selected. The Softmax function will convert all attention scores of the noisy action after the corresponding embedding time step into probabilities between 0 and 1, and the sum of all probabilities is 1. This probability is called attention weight.

[0045] In an optional embodiment, before inputting the reaction action sequence, the robot's current state, time step, and randomly generated noisy action sequence into the action prediction model, the method includes: obtaining an action sample sequence and corresponding robot state samples and action sequence labels; inputting the action sample sequence and corresponding robot state samples and action sequence labels into the model to be trained, so as to positively add noise to each action in the action sequence labels to obtain a noisy action sequence, and extracting features from the input action sample sequence and robot state samples respectively, so as to combine the extracted action feature sequence and state feature sequence to perform reverse denoising on the noisy action sequence to obtain a denoised action sequence; constructing a loss function based on the denoised action sequence and corresponding action sequence labels, and obtaining an action prediction model for generating predicted actions based on the convergence of the loss function.

[0046] It should be added that the action sample sequences are obtained by acquiring video training data and using steps S11-S12, which will not be repeated here. Furthermore, the video training data can be obtained according to actual training needs, such as internet data or existing public datasets like Inter-X, etc., without further limitation here.

[0047] In an alternative embodiment, continue to refer to Figure 2After receiving the predicted action from the action prediction model, the process includes: sending the predicted action to the robot's controller to enable autonomous response and update the robot's state in preparation for the next interaction cycle. Additionally, safety policies can be set to prevent harm to people or the robot itself. These policies can be set based on prior experience and actual design requirements, and are not further limited here.

[0048] In summary, this invention acquires interactive video stream data to capture dynamically changing interaction processes, providing a more comprehensive context for subsequent robot action planning. This makes decision-making safer and more intelligent. It determines the spatial pose information of the interactive object through 3D pose estimation, then combines this with a large language model to understand the interaction intent, ultimately generating a context-appropriate, human-like sequence of robot responses. Furthermore, it combines the robot's current state, time step, and randomly generated noisy action sequence, using an action prediction model to denoise the noisy action sequence based on the robot's current state and response sequence, generating a clear, coherent, and high-quality final output. This ensures that the final predicted action is the result of optimization under global constraints, satisfying interaction requirements while guaranteeing robot safety and task feasibility. It avoids issues such as stiff, jittery, or unnatural robot movements, resulting in a smoother, more human-like final robot performance and improved user experience.

[0049] The robot interaction device provided in the embodiments of this application is described below. The robot interaction device described below and the robot interaction method described above can be referred to in correspondence.

[0050] Figure 3 A schematic diagram of a robot interaction device is shown, the device comprising: Data acquisition module 31 is used to: acquire interactive video stream data; The first action prediction module 32 is used to: perform three-dimensional pose estimation on the interactive object in the interactive video stream data based on the interactive video stream data, determine the spatial pose information, and combine it with a preset large language model to identify the interactive intent and generate a reaction action sequence. The second action prediction module 33 is used to: input the reaction action sequence, the robot's current state, the time step, and the randomly generated noisy action sequence into the action prediction model to obtain the predicted action output by the action prediction model; wherein, the action prediction model is trained based on the action sample sequence and the robot state sample and action sequence label corresponding to the action sample sequence, and the action prediction model is used to perform reverse denoising on the randomly generated noisy action sequence based on the reaction action sequence and the robot's current state to obtain the predicted action.

[0051] In this embodiment, the first motion prediction module 32 includes: a pose estimation unit, used to perform three-dimensional pose estimation on the interactive object in the interactive video stream data based on the interactive video stream data, and obtain the three-dimensional skeletal joint coordinates of the interactive object as pose information; a position determination unit, used to determine the position and orientation of the interactive object relative to the robot based on the pose information and the robot's position coordinates, and obtain spatial position information; a text generation unit, used to generate scene description text based on the spatial position information and a preset text template; and a data fusion unit, used to fuse the pose information, spatial position information and scene description text to obtain spatial pose information.

[0052] It should be added that the data fusion unit includes: a processing subunit, which is used to tokenize the attitude information, spatial location information and scene description text respectively to obtain the corresponding attitude token sequence, spatial location token sequence and text token sequence; and a fusion subunit, which is used to concatenate the attitude token sequence, spatial location token sequence and text token sequence according to a preset concatenation order or a preset combination order to obtain spatial pose information.

[0053] Furthermore, the processing subunit is also used to: normalize the coordinates of the three-dimensional skeletal joints in the posture information, and reduce the dimensionality of the normalized posture information to obtain posture dimensionality-reduced data; encode the posture dimensionality-reduced data to obtain a continuous posture vector with the same dimension as the posture dimensionality-reduced data; quantize the continuous posture vector to obtain the index of the quantized posture vector and determine the posture token; wherein, the quantized posture vector is the posture code vector that is closest to the continuous posture vector in the first codebook obtained earlier.

[0054] It should be added that the data fusion unit also includes a training subunit, which trains the VQ-VAE before determining the code vector that is closest to the continuous pose in the preset codebook based on the continuous pose vector. Specifically, the training subunit is used for: acquiring pose samples; normalizing the pose samples and reducing their dimensionality to obtain dimensionality-reduced pose samples; encoding the dimensionality-reduced pose samples to map them to the latent space to obtain continuous sample vectors; determining the code vector closest to the continuous sample vectors in a pre-defined codebook based on the continuous sample vectors; obtaining discrete pose tokens based on the indices corresponding to the determined code vectors, and generating discrete code vectors based on the determined code vectors and continuous sample vectors; decoding the discrete code vectors to reconstruct them back into the dimensionality-reduced pose samples to obtain decoded vectors; constructing a reconstruction loss function based on the decoded vectors and dimensionality-reduced pose samples; constructing a codebook loss function based on the discrete code vectors and the determined code vectors; and constructing a commitment loss function based on the continuous sample vectors and discrete code vectors; obtaining the total loss function based on the reconstruction loss function, codebook loss function, and commitment loss function; and converging based on the total loss function to terminate training and obtain the VQ-VAE.

[0055] In addition, the processing subunit is also used to: normalize the spatial location information and reduce the dimensionality of the normalized spatial location information to obtain spatial location dimensionality-reduced data; encode the spatial location dimensionality-reduced data to obtain a continuous spatial location vector with the same dimension as the spatial location dimensionality-reduced data; quantize the continuous spatial location vector to obtain the index of the quantized spatial location vector and determine the spatial location token; wherein, the quantized spatial location vector is the spatial location code vector that is closest to the continuous spatial location vector in the previously obtained second codebook.

[0056] In addition, the processing subunit is also used for: SA, constructing a vocabulary by using all characters in the scene description text as the basic units in the initial vocabulary; SB, traversing the scene description text, counting the frequency of all character pairs formed by adjacent characters, and merging the character pair with the highest frequency to obtain a new subword; SC, updating the vocabulary with the new subword, replacing the corresponding character pair with the highest frequency in the scene description text with the new subword to obtain the updated text, and iteratively executing steps SB-SC until a preset stopping condition is reached; SD, dividing the final updated text into subword tokens based on the final vocabulary to obtain a text token sequence.

[0057] In this embodiment, the second action prediction module 33 is further configured to: input the reaction action sequence, the robot's current state, the time step, and the randomly generated noisy action sequence into the action prediction model, so as to embed the time step into each noisy action of the noisy action sequence; according to each noisy action after embedding the time step, use a self-attention mechanism to capture the context information of each noisy action after embedding the time step, and combine the reaction action sequence and the robot's current state to perform reverse denoising on the corresponding noisy action after embedding the time step, to obtain the corresponding denoised action; according to each denoised action, use a self-attention mechanism to capture the context information of each denoised action, and combine the reaction action sequence and the robot's current state to perform reverse denoising on the corresponding denoised action, and iteratively execute this step to gradually denoise the corresponding denoised action until a preset number of iterations is reached, to obtain the predicted action output by the action prediction model.

[0058] Furthermore, the second action prediction module 33 is also used to: determine the query vector, key vector, and value vector of the noisy action after each embedding time step using a self-attention mechanism; for each noisy action after embedding time step, perform a dot product operation on the query vector of the noisy action after embedding time step and the key vector of the noisy action after each embedding time step to determine the corresponding attention score, and normalize all the obtained attention scores to obtain the attention weight of the noisy action after embedding time step; and perform a weighted summation of the attention weight of the noisy action after embedding time step and the value vector of the noisy action after embedding time step to obtain the context information of the noisy action after embedding time step.

[0059] In an optional embodiment, the device further includes: a training module, configured to: obtain an action sample sequence and corresponding robot state samples and action sequence labels before inputting the reaction action sequence, robot current state, time step, and randomly generated noisy action sequence into the action prediction model; input the action sample sequence and corresponding robot state samples and action sequence labels into the model to be trained, to positively add noise to each action in the action sequence labels to obtain a noisy action sequence, and to extract features from the input action sample sequence and robot state samples respectively, to combine the extracted action feature sequence and state feature sequence to perform reverse denoising on the noisy action sequence to obtain a denoised action sequence; construct a loss function based on the denoised action sequence and corresponding action sequence labels, and obtain an action prediction model for generating predicted actions based on the convergence of the loss function.

[0060] In an optional embodiment, the device further includes: an action execution module, which, after obtaining the predicted action output by the action prediction model, sends the predicted action to the robot's controller to complete an autonomous response based on the predicted action and update the robot's state in preparation for the next interaction cycle.

[0061] In summary, this embodiment of the invention acquires interactive video stream data through a data acquisition module to capture dynamically changing interaction processes, providing a more comprehensive context for subsequent robot action planning, making decision-making safer and more intelligent. A first action prediction module performs 3D pose estimation to determine the spatial pose information of the interactive object, then combines this with a large language model to understand its interaction intent, ultimately generating a context-appropriate, human-like robot response action sequence. Furthermore, a second action prediction module combines the robot's current state, time step, and randomly generated noisy action sequence, using an action prediction model to denoise the noisy action sequence based on the robot's current state and response action sequence, generating a clear, coherent, and high-quality final output. This ensures that the final predicted action is the result of optimization under global constraints, meeting interaction requirements while guaranteeing robot safety and task feasibility, avoiding stiff, jittery, or unnatural robot movements, resulting in a smoother, more human-like robot performance and improved user experience.

[0062] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 410, a communication interface 420, a memory 430, and a communication bus 440. The processor 410, communication interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call a computer program in the memory 430 to execute steps of a robot interaction method, such as: acquiring interactive video stream data; performing three-dimensional pose estimation on the interactive object in the interactive video stream data based on the interactive video stream data, determining spatial pose information, and recognizing the interaction intent using a preset large language model to generate a response action sequence; inputting the response action sequence, the robot's current state, time step, and a randomly generated noisy action sequence into a motion prediction model to obtain the predicted action output by the motion prediction model; wherein the motion prediction model is trained based on the action sample sequence and the corresponding robot state sample and action sequence label, and is used to perform reverse denoising on the randomly generated noisy action sequence based on the response action sequence and the robot's current state to obtain the predicted action.

[0063] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0064] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the robot interaction method provided in the above embodiments, such as: acquiring interactive video stream data; performing three-dimensional pose estimation on the interactive object in the interactive video stream data based on the interactive video stream data, determining spatial pose information, and recognizing the interaction intention in combination with a preset large language model to generate a reaction action sequence; inputting the reaction action sequence, the robot's current state, time step, and randomly generated noisy action sequence into the action prediction model to obtain the predicted action output by the action prediction model; wherein, the action prediction model is trained based on the action sample sequence and the robot state sample and action sequence label corresponding to the action sample sequence, and the action prediction model is used to perform reverse denoising on the randomly generated noisy action sequence based on the reaction action sequence and the robot's current state to obtain the predicted action.

[0065] On the other hand, embodiments of this application also provide a processor-readable storage medium storing a computer program. The computer program is used to cause a processor to execute the steps of the robot interaction methods provided in the above embodiments, including, for example: acquiring interactive video stream data; performing three-dimensional pose estimation on the interactive object in the interactive video stream data based on the interactive video stream data, determining spatial pose information, and recognizing the interaction intent by combining a preset large language model to generate a reaction action sequence; inputting the reaction action sequence, the robot's current state, time step, and a randomly generated noisy action sequence into a motion prediction model to obtain the predicted action output by the motion prediction model; wherein the motion prediction model is trained based on the action sample sequence and the corresponding robot state sample and action sequence label, and the motion prediction model is used to perform reverse denoising on the randomly generated noisy action sequence based on the reaction action sequence and the robot's current state to obtain the predicted action.

[0066] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0067] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0068] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A robot interaction method, characterized in that, include: Acquire interactive video stream data; Based on the interactive video stream data, the three-dimensional pose of the interactive object in the interactive video stream data is estimated to determine the spatial pose information, and the interactive intention is identified by combining the preset large language model to generate a reaction action sequence. The reaction action sequence, the robot's current state, the time step, and the randomly generated noisy action sequence are input into the action prediction model to obtain the predicted action output by the action prediction model. The action prediction model is trained based on the action sample sequence and the corresponding robot state sample and action sequence label. The action prediction model is used to perform reverse denoising on the randomly generated noisy action sequence based on the reaction action sequence and the robot's current state to obtain the predicted action.

2. The robot interaction method according to claim 1, characterized in that, The reaction action sequence, the robot's current state, the time step, and the randomly generated noisy action sequence are input into the action prediction model to obtain the predicted action output by the action prediction model, including: The reaction action sequence, the robot's current state, the time step, and the randomly generated noisy action sequence are input into the action prediction model to embed the time step into each noisy action of the noisy action sequence. Based on the noisy actions embedded after each time step, the context information of the noisy actions embedded after each time step is captured using a self-attention mechanism. Combined with the reaction action sequence and the robot's current state, the noisy actions embedded after the corresponding time step are de-denoised to obtain the corresponding de-denoised actions. Based on each of the denoising actions, the contextual information of each denoising action is captured using a self-attention mechanism. Combined with the reaction action sequence and the robot's current state, the corresponding denoising action is reversed and denoised. This step is iteratively executed to gradually denoise the corresponding denoising action until a preset number of iterations is reached, thereby obtaining the predicted action output by the action prediction model.

3. The robot interaction method according to claim 1, characterized in that, Before inputting the reaction action sequence, the robot's current state, time step, and randomly generated noisy action sequence into the action prediction model, the following steps are included: Obtain action sample sequences, as well as corresponding robot state samples and action sequence labels; The action sample sequence, the robot state sample corresponding to the action sample sequence, and the action sequence label are input into the model to be trained. Each action in the action sequence label is positively denoised to obtain a noisy action sequence. Features are extracted from the input action sample sequence and robot state sample respectively. The extracted action feature sequence and state feature sequence are combined to perform reverse denoising on the noisy action sequence to obtain a denoised action sequence. Based on the denoised action sequence and the corresponding action sequence label, a loss function is constructed, and based on the convergence of the loss function, an action prediction model for generating predicted actions is obtained.

4. The robot interaction method according to claim 1, characterized in that, Based on the interactive video stream data, perform 3D pose estimation on the interactive objects in the interactive video stream data to determine spatial pose information, including: Based on the interactive video stream data, the three-dimensional pose estimation of the interactive object in the interactive video stream data is performed to obtain the three-dimensional skeletal joint coordinates of the interactive object as pose information. Based on the posture information and the robot's position coordinates, the position and orientation of the interactive object relative to the robot are determined, thus obtaining spatial position information; Based on the spatial location information and a preset text template, a scene description text is generated; The spatial pose information is obtained by fusing the posture information, the spatial location information, and the scene description text.

5. The robot interaction method according to claim 4, characterized in that, The spatial pose information is obtained by fusing the posture information, the spatial location information, and the scene description text, including: The attitude information, the spatial location information, and the scene description text are tokenized respectively to obtain the corresponding attitude token sequence, spatial location token sequence, and text token sequence. The spatial pose information is obtained by concatenating the pose token sequence, the spatial location token sequence, and the text token sequence in a preset concatenation order or a preset combination order.

6. The robot interaction method according to claim 5, characterized in that, The gesture information is tokenized, including: The coordinates of the three-dimensional skeletal joints in the posture information are normalized, and the normalized posture information is then reduced in dimensionality to obtain posture dimensionality-reduced data. The attitude dimensionality reduction data is encoded to obtain a continuous attitude vector with the same dimension as the attitude dimensionality reduction data. The continuous attitude vector is quantized to obtain the index of the quantized attitude vector, and the attitude token is determined; wherein, the quantized attitude vector is the attitude code vector that is closest to the continuous attitude vector in the first codebook obtained earlier; The spatial location information is tokenized, including: The spatial location information is normalized, and the normalized spatial location information is then dimensionality reduced to obtain dimensionality-reduced spatial location data. The spatial location dimensionality reduction data is encoded to obtain a continuous spatial location vector with the same dimension as the spatial location dimensionality reduction data. The continuous spatial location vector is quantized to obtain the index of the quantized spatial location vector, and the spatial location token is determined; wherein, the quantized spatial location vector is the spatial location code vector that is closest to the continuous spatial location vector in the previously obtained second codebook; The scene description text is tokenized, including: SA uses all the characters in the scene description text as the basic units in the initial vocabulary to construct a vocabulary; SB, iterate through the scene description text, count the frequency of all character pairs consisting of adjacent characters, and select the character pair with the highest frequency to merge and obtain a new subword; SC updates the vocabulary with the new subword, replaces the most frequent character pair in the scene description text with the new subword to obtain the updated text, and iterates through steps SB-SC until a preset stopping condition is reached. SD, based on the final vocabulary, segments the final updated text into sub-word tokens, resulting in a text token sequence.

7. The robot interaction method according to claim 1, characterized in that, After obtaining the predicted action output by the action prediction model, the following steps are included: The predicted action is sent to the robot's controller so that the robot can perform an autonomous response and update its state based on the predicted action.

8. A robot interaction device, characterized in that, include: The data acquisition module is used to acquire interactive video stream data. The first action prediction module is used to: perform three-dimensional pose estimation on the interactive object in the interactive video stream data based on the interactive video stream data, determine the spatial pose information, and combine it with a preset large language model to identify the interactive intention and generate a reaction action sequence. The second action prediction module is used to: input the reaction action sequence, the robot's current state, the time step, and the randomly generated noisy action sequence into the action prediction model to obtain the predicted action output by the action prediction model; wherein, the action prediction model is trained based on the action sample sequence and the robot state sample and action sequence label corresponding to the action sample sequence, and the action prediction model is used to perform reverse denoising on the randomly generated noisy action sequence based on the reaction action sequence and the robot's current state to obtain the predicted action.

9. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the robot interaction method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the robot interaction method according to any one of claims 1 to 7.

Images