Multi-mode tool body intelligent control method and device and electronic equipment

Abstract

The invention provides a multi-mode intelligent control method and device for a robot body and electronic equipment. The multi-mode intelligent control method comprises the following steps: acquiring multi-view images and task texts of cameras at multiple parts of the robot; generating two-dimensional track supervision information and key point supervision information based on the two, and converting the two-dimensional track supervision information and the key point supervision information into a visual supervision graph; key point features are generated through the supervision graph, and track features are generated in combination with the multi-view image; extracting a depth image, optical flow information, a matched segmentation mask, an execution mechanism attitude and a motion frequency, and generating corresponding features; inputting each feature into a multi-modal fusion reasoning model, generating a relative action prediction feature and converting the relative action prediction feature into an action control instruction; and controlling an execution mechanism to complete the task. According to the method, track supervision, key point supervision and multi-modal prior are introduced under the vision-language condition, and perception integrity, robustness and execution precision are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence and robot control, in particular, to a multi-modal embodied intelligence control method and device and electronic equipment. BACKGROUND

[0002] Embodied intelligence aims to realize autonomous interaction of machines with real environments through a closed loop of perception, understanding and action. In the field of robot control, the fusion of multi-modal inputs (such as vision, language, depth, state sensing, etc.) is one of the core directions to improve execution accuracy and task generalization ability.

[0003] Existing multi-modal robot control methods usually directly input images and task texts into a control strategy model, and only rely on the final execution action as a supervision signal during the training process. The model is difficult to learn an effective mapping from high-dimensional perception information to the action space, resulting in a lack of controllability and transparency in action planning. In tasks such as grasping, plugging, and assembly that require millimeter-level precision, the model is easily affected by cumulative errors, leading to a high task failure rate. At the same time, the generalization and transfer ability is limited, and when the training data is insufficient or the execution environment changes (light, occlusion, multi-view differences, etc.), the model performance decreases significantly. SUMMARY

[0004] The embodiments of the present disclosure at least provide a multi-modal embodied intelligence control method, device and electronic equipment, which introduces trajectory supervision, key point supervision and multi-modal prior feature in the vision-language condition, and can more stably learn the mapping relationship from multi-modal perception to fine action, significantly improve the integrity and robustness of task perception, and have higher execution accuracy.

[0005] The embodiments of the present disclosure provide a multi-modal embodied intelligence control method, comprising:

[0006] obtaining multi-view images captured by cameras at multiple positions of a robot, and a task text;

[0007] generating two-dimensional trajectory supervision information and key point supervision information based on the task text and the multi-view images, and converting the two-dimensional trajectory supervision information and the key point supervision information into a visualized supervision graph;

[0008] generating key point features corresponding to the key point supervision information according to the visualized supervision graph, and generating corresponding trajectory features and key point features in combination with the multi-view images;

[0009] determining a depth image, optical flow information, a segmentation mask matched with the task text corresponding to the multi-view images, and posture information and motion frequency information of an execution mechanism, and respectively generating corresponding depth features, optical flow features, segmentation features, posture features and frequency features;

[0010] inputting the trajectory feature, the key point feature, the depth feature, the optical flow feature, the segmentation feature, the pose feature and the frequency feature into a multi-modal fusion inference model together to generate relative action prediction features and convert them into action control instructions;

[0011] controlling a robot execution mechanism to complete a preset task according to the action control instructions.

[0012] In an optional implementation, based on the task text and the multi-view image, two-dimensional trajectory supervision information and key point supervision information are generated, specifically including:

[0013] image information from a left arm camera, a right arm camera and a head camera of the robot in the multi-view image is input into an image encoder and linearly projected to obtain corresponding image visual features;

[0014] the task text is input into a text encoder to obtain task text features, and multi-modal fusion is performed on the image visual features and the task text features, and the two-dimensional trajectory supervision information and the key point supervision information of the text modal are output;

[0015] wherein, the two-dimensional coordinates and execution states of a robot end effector in a preset view image coordinate system are extracted to generate trajectory sequence information of the text modal as the two-dimensional trajectory supervision information;

[0016] entity words of an object being operated are parsed from the task text, and key point positions of the object being operated and the robot end effector are predicted to generate key point coordinates and key point names of the text modal as the key point supervision information.

[0017] In an optional implementation, the two-dimensional trajectory supervision information and the key point supervision information are converted into a visual supervision graph, specifically including:

[0018] a trajectory supervision image is generated based on a robot head camera view, containing a two-dimensional trajectory path and corresponding execution state of a robot end effector according to the two-dimensional trajectory supervision information;

[0019] a key point supervision image is generated based on the robot head camera view, containing corresponding key point positions of an object being operated and the robot end effector according to the key point supervision information.

[0020] In an optional implementation, for the trajectory supervision image, a starting point, a path and an end point in the two-dimensional trajectory path are color-coded differently, and a preset pattern is used to label corresponding action nodes of the robot end effector in the two-dimensional trajectory path, and different execution states are distinguished by color.

[0021] According to the key point position, the preset pattern is used to label the corresponding key points of the object being operated and the robot end effector of the key point supervision image, and color coding is performed.

[0022] In an optional implementation, according to the visual supervision graph, the key point features corresponding to the key point supervision information are generated, and the trajectory features and the key point features corresponding to the multi-view images are combined, specifically including:

[0023] The key point supervision image is subjected to convolution operation and linear projection to extract the key point features;

[0024] After the trajectory supervision image is multiplied by a preset first coefficient and the image information collected by the robot head camera is multiplied by a preset second coefficient, different transparency fusion is used to generate a global trajectory running image of the robot head view;

[0025] The global trajectory running image is subjected to convolution operation and linear projection to generate the trajectory features.

[0026] In an optional implementation, the optical flow features are generated based on the following steps:

[0027] For each view camera image, the starting camera image and the ending camera image of the trajectory sequence under the view are extracted, and the optical flow information between the starting camera image and the ending camera image under the view is determined;

[0028] The optical flow information under each view is subjected to re-projection to extract projection features, and the projection features under the robot left arm view and the robot right arm view are subjected to cross-attention operation with the projection features under the robot head view, respectively, and then the cross-attention result features are spliced to determine the optical flow features.

[0029] In an optional implementation, the model is trained based on the following steps:

[0030] Sample operation video data of the robot execution mechanism is obtained, and corresponding text is configured for the execution action corresponding to the sample operation video data;

[0031] The motion posture of the robot execution mechanism is labeled as a preset requirement label and a key point label, and an image feature extraction model corresponding to the multi-view image is trained according to the preset requirement label and the key point label;

[0032] After the image feature extraction model is trained, the image feature extraction model is frozen, and the multi-modal fusion inference model is trained, so that the multi-modal fusion inference model has the ability to output relative action prediction features.

[0033] The embodiment of the present disclosure further provides a multi-modal embodied intelligent control device, comprising:

[0034] a data acquisition module, configured to acquire multi-view images captured by cameras at multiple positions of a robot, and task text;

[0035] a trajectory and key point supervision module, configured to generate two-dimensional trajectory supervision information and key point supervision information based on the task text and the multi-view images, and convert the two-dimensional trajectory supervision information and the key point supervision information into a visual supervision graph;

[0036] a first feature extraction module, configured to generate key point features corresponding to the key point supervision information according to the visual supervision graph, and generate corresponding trajectory features and key point features in combination with the multi-view images;

[0037] a second feature extraction module, configured to determine a depth image, optical flow information, and a segmentation mask matched with the task text corresponding to the multi-view images, and determine posture information and motion frequency information corresponding to an execution mechanism, and generate corresponding depth features, optical flow features, segmentation features, posture features, and frequency features;

[0038] an action prediction module, configured to input the trajectory features, the key point features, the depth features, the optical flow features, the segmentation features, the posture features, and the frequency features into a multi-modal fusion inference model, generate relative action prediction features, and convert the relative action prediction features into an action control instruction;

[0039] a control execution module, configured to control the robot execution mechanism to complete a preset task according to the action control instruction.

[0040] The embodiment of the present disclosure further provides an electronic device, comprising a processor, a memory, and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform the multi-modal embodied intelligent control method or the steps in any possible implementation manner of the multi-modal embodied intelligent control method.

[0041] The embodiment of the present disclosure further provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the multi-modal embodied intelligent control method or the steps in any possible implementation manner of the multi-modal embodied intelligent control method.

[0042] The embodiment of the present disclosure further provides a computer program product comprising computer programs / instructions which, when executed by a processor, implement the multi-modal embodiment intelligent control method or the steps in any possible implementation of the multi-modal embodiment intelligent control method.

[0043] The embodiment of the present disclosure provides a multi-modal embodiment intelligent control method, device and electronic equipment. A plurality of multi-view images captured by cameras at a plurality of positions of a robot are obtained, as well as a task text. Based on the task text and the multi-view images, two-dimensional trajectory supervision information and key point supervision information are generated and converted into a visual supervision graph. According to the visual supervision graph, key point features corresponding to the key point supervision information are generated, and corresponding trajectory features and key point features are generated in combination with the multi-view images. Depth images, optical flow information, segmentation masks matched with the task text, and posture information and motion frequency information of an executing mechanism corresponding to the multi-view images are determined, and corresponding depth features, optical flow features, segmentation features, posture features and frequency features are generated. The trajectory features, the key point features, the depth features, the optical flow features, the segmentation features, the posture features and the frequency features are jointly input into a multi-modal fusion inference model to generate relative action prediction features and convert them into action control instructions. The robot executing mechanism is controlled to complete a preset task according to the action control instructions. The embodiment intelligent method introduced under the visual-language condition can more stably learn the mapping relationship from multi-modal perception to fine action, significantly improve the integrity and robustness of task perception, and has higher execution accuracy.

[0044] In order to make the above objectives, characteristics and advantages of the present disclosure more apparent and understandable, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings needed to be used in the embodiments. The drawings herein are incorporated into the specification and form a part of the specification. These drawings show embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be considered as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0046] Figure 1 A flowchart of a multi-modal embodiment intelligent control method provided by the embodiment of the present disclosure is shown;

[0047] Figure 2A flowchart of a model training method is shown.

[0048] Figure 3 A schematic diagram of a multi-modal embodied intelligent control device is shown.

[0049] Figure 4 A schematic diagram of an electronic device is shown. DETAILED DESCRIPTION

[0050] To make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the following will be combined with the accompanying drawings for the embodiments of the present disclosure to make a clear and complete description of the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The components of the embodiments of the present disclosure described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed present disclosure, but only represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present disclosure.

[0051] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0052] The term "and / or" herein only describes an association relationship, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B and C can mean including any one or more elements selected from the set consisting of A, B and C.

[0053] It is found through research that the existing multi-modal robot control method usually directly inputs images and task text into a control strategy model, and only relies on the final execution action as a supervision signal in the training process. The model is difficult to learn an effective mapping from high-dimensional perception information to the action space, resulting in a lack of controllability and transparency in action planning. In tasks such as grasping, plugging, assembly, etc. that require millimeter-level precision, the model is easily affected by cumulative errors, resulting in a high task failure rate. At the same time, the generalization and migration ability is limited, and when the training data is insufficient or the execution environment changes (light, occlusion, multi-view difference, etc.), the model performance decreases significantly.

[0054] Based on the above research, the present disclosure provides a multi-modal embodied intelligent control method, device and electronic equipment. A plurality of multi-view images captured by cameras at multiple parts of a robot are obtained, as well as a task text. Based on the task text and the multi-view images, two-dimensional trajectory supervision information and key point supervision information are generated and converted into a visual supervision graph. According to the visual supervision graph, key point features corresponding to the key point supervision information are generated, and corresponding trajectory features and key point features are generated in combination with the multi-view images. The depth images, optical flow information, segmentation mask matched with the task text, and posture information and motion frequency information of the actuator corresponding to the multi-view images are determined, and corresponding depth features, optical flow features, segmentation features, posture features and frequency features are generated. The trajectory features, the key point features, the depth features, the optical flow features, the segmentation features, the posture features and the frequency features are jointly input into a multi-modal fusion inference model to generate relative action prediction features and convert them into action control instructions. The robot actuator is controlled to complete a preset task according to the action control instructions. The embodied intelligent method introduced under the visual-linguistic condition with trajectory supervision, key point supervision and multi-modal prior features can more stably learn the mapping relationship from multi-modal perception to fine action, significantly improve the integrity and robustness of task perception, and has higher execution accuracy.

[0055] To facilitate the understanding of the present embodiment, first, a multi-modal embodied intelligent control method disclosed by the present embodiment is introduced in detail. The execution subject of the multi-modal embodied intelligent control method provided by the present embodiment is generally a computer device with certain computing power, which may, for example, include a terminal device or a server or other processing device. The terminal device may be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementation manners, the multi-modal embodied intelligent control method may be realized by a processor calling computer readable instructions stored in a memory.

[0056] Referring to Figure 1 The flowchart of the multi-modal embodied intelligent control method provided by the present embodiment is shown. The method includes steps S101-S105.

[0057] S101, a plurality of multi-view images captured by cameras at multiple parts of a robot are obtained, as well as a task text.

[0058] In a specific embodiment of the present application, a plurality of multi-view images captured by cameras at multiple positions of the robot and a task text are first acquired to achieve comprehensive perception and semantic understanding of the task scene. The robot comprises a plurality of camera groups arranged at positions of the left arm, the right arm and the head. Preferably, the left arm and right arm cameras are respectively installed at the end of the mechanical arm or near the joints to collect detail images of the operation area at close range, which can accurately capture the object state, posture change and local occlusion around the end effector. The head camera is installed at a high position of the robot body to acquire global scene images, which can provide overall spatial layout information and environmental background information during task execution.

[0059] Here, to ensure the consistency of the multi-view images in time sequence, a unified time synchronization mechanism is adopted in the data acquisition process in this embodiment. Through the way of hardware trigger signal or software timestamp, the image frames from each camera are synchronously acquired at the same time point, thereby ensuring the spatio-temporal consistency of the multi-view images in fusion and comparative analysis.

[0060] The task text is generated by an external interactive system and can be derived from natural language instructions of a user, a preset task script or scene label information, and specifically includes descriptions of the operation object, operation mode, target position and constraint conditions of the target task. For example, in the execution of the task of "putting the red cup on the right side of the table", the task text will contain entity and action description information such as "red cup", "put", "right side of the table", etc.

[0061] In this way, by simultaneously inputting the task text and the multi-view images into the subsequent multi-modal processing module, the robot can establish a corresponding relationship between the semantic context of the task target and the environmental visual information in the perception stage.

[0062] It should be noted that in the acquisition process, the multi-view images are not limited to color images, but can also include depth maps collected by the head camera or other depth sensors to provide spatial geometric structure information of the scene to assist subsequent three-dimensional position reasoning and path planning. In order to improve the quality of image data, the system can perform automatic exposure, white balance and noise suppression preprocessing operations before acquisition to reduce the influence of light changes, color distortion and sensor noise on subsequent feature extraction. In some application scenarios, an image distortion correction algorithm can also be introduced to correct the geometric distortion caused by wide-angle or fisheye lenses, thereby ensuring the coordinate accuracy of trajectory supervision and key point prediction.

[0063] S102, based on the task text and the multi-view images, generating two-dimensional trajectory supervision information and key point supervision information, and converting into a visual supervision graph.

[0064] In specific implementation, based on the task text and multi-view images, the process of generating two-dimensional trajectory supervision information and key point supervision information introduces strong explainable intermediate supervision signals for multi-modal embodied intelligent control, and abstract paths and target information are converted into image representations that can be directly perceived by the model through visual forms.

[0065] Specifically, first, multi-view images from the left and right arms of the robot and the head camera are input into an image encoding module, which preferably uses a visual encoder with cross-scene generalization capability, and the high-dimensional feature vectors output by the module are mapped to a unified feature space through linear projection, so as to be subsequently fused with task text features in a multi-modal manner. In terms of text processing, the task text is input into a text encoder to extract task semantic information, including the operation object, target position, action type and related constraints, and the task text features are interacted with the image features of each view through a multi-modal fusion network to generate two types of textual prediction results, namely two-dimensional trajectory supervision information and key point supervision information.

[0066] Here, the two-dimensional trajectory supervision information is used to describe the two-dimensional path trajectory and the corresponding execution state of the robot end effector under a specific camera view during task execution. This embodiment preferably takes the head camera view as the reference coordinate system, extracts the two-dimensional coordinate points of the end effector in consecutive time frames, and combines the opening and closing state of the gripper, the speed change and other execution state parameters to express these information in the form of text sequences. For example, the trajectory sequence information can include the (x, y) coordinates of each path point and the corresponding execution state label, forming a set of path-state pairs. The key point supervision information is used to represent the spatial relationship between the task target object and the end effector, by analyzing the entity words in the task text, identifying the category of the object being operated, and predicting the positions of the key points related to the end effector, such as the grasping point, the support point or the placement point, and finally outputting in the form of key point coordinates and names.

[0067] After obtaining the above two types of supervision information, they are respectively converted into visual supervision maps to enhance the spatial intuitive perception ability of the model. Specifically, for the trajectory supervision information, according to the two-dimensional path of the end effector under the head perspective, a trajectory supervision image containing the complete motion trajectory is generated, wherein the starting point, the middle section of the path and the end point of the trajectory are distinguished by different color coding, and the gripper execution state is embodied by a preset pattern or color mark at the trajectory node, so that the action sequence and state change are visually clear. For the key point supervision information, the predicted key point positions are labeled on the image with the head camera perspective as the background, different colors and graphical symbols are used to distinguish the object key points and the end effector key points, and text labels or color coding are added near the key points to represent their semantic categories. Both the trajectory supervision image and the key point supervision image use a black background or a task scene image as the base to ensure that the trajectory and key point areas can be accurately focused on during subsequent feature extraction, reducing background interference.

[0068] As a possible implementation, the image information from the left arm, right arm and head camera of the robot in the multi-view image is respectively input into the image encoder and linearly projected to obtain corresponding image visual features; the task text is input into the text encoder to obtain the task text features, and the image visual features and the task text features are multi-modal fused to output the two-dimensional trajectory supervision information and the key point supervision information of the text modality; wherein the two-dimensional coordinates and execution state of the robot end effector under the preset perspective image coordinate system are extracted to generate the trajectory sequence information of the text modality as the two-dimensional trajectory supervision information; the entity words of the operated object are parsed from the task text, and the key point positions of the operated object and the robot end effector are predicted to generate the key point coordinates and the key point names of the text modality as the key point supervision information.

[0069] In specific implementation, the image information from the left arm, right arm and head camera of the robot in the multi-view image is respectively input into the image encoder for feature extraction. After the feature extraction is completed, the high-dimensional features under different camera perspectives are mapped to a unified feature space via a linear projection layer to ensure the consistency of the features in terms of dimension and distribution, facilitating subsequent multi-modal fusion processing.

[0070] Here, the image features of the left arm and right arm cameras can focus on capturing fine-grained texture information and local geometric changes of the operation area, while the image features of the head camera provide global layout and environmental context information of the task scene. Through this complementary perspective, accurate spatial perception can be achieved in subsequent fusion.

[0071] Meanwhile, the task text is input into a text encoder to extract task semantic information. The text encoder can be a pre-trained language model such as T5, BERT, Gemma, etc. Through tokenization, embedding and context modeling, task text features capable of representing task intent, target object, action type and location constraint, etc. are obtained. After obtaining the image visual features and task text features, the two are input into a multi-modal fusion module, so that the text features can selectively focus on the task-related areas and details in the visual features from different perspectives, thereby generating a more task-oriented fusion representation.

[0072] Further, based on the fusion result, two types of text modal prediction information are output, namely two-dimensional trajectory supervision information and key point supervision information. The generation process of the two-dimensional trajectory supervision information includes: in the preset perspective image coordinate system, the two-dimensional coordinate positions of the robot end effector in the continuous time frames are extracted, and the execution state information such as the opening and closing state of the gripper, the speed change, the grasping or placing action mark, etc. is recorded at the same time. These time series coordinates and states are combined to form trajectory sequence information and are output in a textual manner. The generation process of the key point supervision information includes: first, the entity words of the operated object are parsed from the task text, such as "cup", "screw", "button", etc. Then, the key point positions related to the object and the key point relationships between the object and the end effector, such as the grasping point, the contact surface, the insertion port, etc. are predicted in the fusion features. Finally, the two-dimensional coordinates of the key points and their corresponding semantic names are output as text modal information.

[0073] As a possible implementation, according to the two-dimensional trajectory supervision information, a trajectory supervision image is generated based on the perspective of the robot head camera, which contains a two-dimensional trajectory path and the corresponding execution state of the robot end effector. According to the key point supervision information, a key point supervision image is generated based on the perspective of the robot head camera, which contains the corresponding key point positions of the operated object and the robot end effector. For the trajectory supervision image, the start point, path and end point in the two-dimensional trajectory path are color-coded, and the corresponding action nodes of the robot end effector are labeled in the two-dimensional trajectory path using a preset pattern, and different execution states are distinguished by color. For the key point supervision image, according to the key point positions, the corresponding key points of the operated object and the robot end effector are labeled using a preset pattern and color-coded.

[0074] In specific implementation, after obtaining the two-dimensional trajectory supervision information and the key point supervision information, first, taking the perspective of the robot head camera as a spatial reference, these information is converted into visual supervision images that can be directly used for visual feature extraction.

[0075] Specifically, for trajectory supervision information, the system draws the trajectory path on a blank background or scene image according to the two-dimensional trajectory sequence of the end effector in the task execution process in the coordinate system of the head camera view, so that the motion trajectory of the effector in the entire task is intuitively reflected.

[0076] Here, in order to enable the trajectory supervision graph to clearly reflect the execution steps and state changes in the subsequent feature learning process, the starting point, intermediate path segment and end point of the trajectory path are encoded with different colors, for example, the starting point is identified with green, the path is identified with gradient color, and the end point is identified with red, thereby visually distinguishing different trajectory stages.

[0077] Meanwhile, a preset pattern mark such as a dot, a triangle symbol or a specific shape is superimposed at the key action node of the trajectory path to represent the execution state of the end effector at the node (such as jaw opening and closing, pause, acceleration, etc.), and different state categories are further distinguished by color or filling method, so that the trajectory supervision graph can carry dual semantics of path information and action state.

[0078] Further, for key point supervision information, the key point positions of the object to be operated and the key point positions related to the end effector are labeled on the key point supervision image based on the view of the head camera of the robot. In order to ensure the distinguishability in the multi-key point scene, the embodiment uses different preset pattern shapes (such as square, diamond, star, etc.) to distinguish the object key points and the effector key points when labeling the key points, and assigns specific color codes to different categories of key points, for example, the key points of the target object are identified with yellow, and the key points of the end effector are identified with purple, thereby visually establishing clear category attribution.

[0079] In this way, through the above visualization conversion process, the embodiment converts the originally abstract trajectory data and key point data into information images with intuitive spatial meaning and semantic hierarchy, which not only retains accurate geometric and state information, but also provides significant intermediate supervision signals for model training. When the supervision image is input into the feature extraction network in the subsequent process, the network can learn the explicit representation of the trajectory and the key point, thereby significantly improving the accuracy and robustness of the control strategy in path planning, target positioning and action execution, etc.

[0080] S103, generating key point features corresponding to the key point supervision information according to the visualized supervision graph, and generating corresponding trajectory features and key point features in combination with the multi-view images.

[0081] In a specific embodiment of the present application, after the generation of the trajectory supervision image and the key point supervision image is completed, feature extraction is first performed on the key point supervision image to obtain the corresponding key point features. Specifically, the key point supervision image is input into a feature extraction module, which preferably includes a convolutional layer, a batch normalization layer, and a nonlinear activation unit. The spatial distribution features and shape semantic information of the key point region are extracted through multi-layer convolutional operations, and the features are mapped to a unified multi-modal fusion space through a linear projection layer after convolutional processing, thereby forming a key point feature vector that can represent the position relationship between the object being operated and the key part of the end effector. The feature vector not only contains the spatial coordinate information of the key point, but also contains the class and functional semantics implied by the color, shape, and other markers, providing a direct basis for subsequent motion planning.

[0082] Here, the key point supervision image is convolved and linearly projected to extract key point features. The trajectory supervision image is multiplied by a preset first coefficient, the image information captured by the robot head camera is multiplied by a preset second coefficient, and then different transparency fusion is used to generate a global trajectory running image from the perspective of the robot head. Convolutional operation and linear projection are performed on the global trajectory running image to generate trajectory features.

[0083] In the process of generating trajectory features, the present embodiment fuses the trajectory supervision image with multi-view images to simultaneously retain trajectory path and environmental background information. Specifically, the trajectory supervision image is multiplied by a preset first coefficient, the corresponding view image captured by the robot head camera is multiplied by a preset second coefficient, and then the two are superimposed with different transparency to generate a global trajectory running image containing global scene background and explicit trajectory markers. The object layout, obstacle position, and environmental texture information in the background can be retained while retaining the trajectory visibility and state markers, thereby providing conditions for joint modeling of the trajectory and the scene.

[0084] Subsequently, the global trajectory running image is input into a convolutional feature extraction network, and the geometric morphology, color coding distribution, and relative position relationship with the environment of the path are extracted through multi-layer convolution and feature downsampling. The trajectory feature vector is obtained by uniformly mapping the output through a linear projection layer to the same feature space as the key point features.

[0085] In addition, in the process of generating trajectory features and key point features in combination with multi-view images, the present embodiment can complementarily fuse features from the left arm, right arm, and head camera to fully utilize close-range operation details and panoramic layout information. For example, the local object texture and motion detail features captured by the left arm and right arm cameras can be fused with the global spatial features provided by the head camera through a cross-view attention mechanism, so that the generated trajectory features and key point features have both fine-grained local accuracy and global spatial consistency.

[0086] This fusion method can effectively improve the model's adaptability to precise trajectory planning and key point positioning in complex task scenarios, and provide high-quality input for subsequent multi-modal fusion reasoning model action prediction.

[0087] In S104, the depth image corresponding to the multi-view image, the optical flow information, the segmentation mask matched with the task text, and the posture information and motion frequency information of the execution mechanism are determined, and corresponding depth features, optical flow features, segmentation features, posture features, and frequency features are generated.

[0088] In one specific embodiment of the present application, after obtaining the trajectory features and key point features, various auxiliary modal features are further extracted from the multi-view images and task-related information to improve the perception accuracy and action stability of the control strategy in a dynamic environment.

[0089] Here, the depth image corresponding to the multi-view image is first determined. The depth image can be directly collected by a head camera with depth perception capability, or calculated by binocular matching, structured light, or depth estimation algorithm from multi-view color images. The depth image can provide spatial distance information of each pixel point in the scene, which is helpful for subsequent establishment of target position and path planning constraints in three-dimensional space. In this embodiment, the depth image is input into a convolutional encoding network, and after multi-layer convolution, pooling and feature compression, multi-scale spatial geometric features are extracted, and are mapped to a unified dimension of depth feature vectors through linear projection, so as to be fused with other modal features.

[0090] Here, in terms of obtaining optical flow information, the starting frame and the end frame of the trajectory sequence are preferably selected for each view, and the optical flow estimation algorithm (such as RAFT, PWC-Net or other high-precision optical flow network) is used to calculate the motion vector field of the inter-frame pixels, and the optical flow map reflecting the dynamic changes of the scene is obtained.

[0091] The optical flow map can explicitly capture the changes in object position and occlusion caused by robot action during task execution, thereby providing a basis for dynamic environment modeling. The optical flow map is projected by features, and the optical flow features under the left arm and right arm view and the optical flow features under the head camera view are cross-view cross-attention fused to obtain fused optical flow features with local fine motion and global dynamic relationship. Finally, the optical flow feature vector is generated through convolution and linear projection.

[0092] Specifically, for each camera image of each view, the starting camera image and the ending camera image of the trajectory sequence under the view are extracted, and the optical flow information between the starting camera image and the ending camera image under the view is determined; the projection features are extracted by re-projecting the optical flow information under each view; the projection features under the left arm and right arm views of the robot are respectively subjected to cross-attention operation with the projection features under the head view of the robot, and then the result features of the cross-attention are spliced to determine the optical flow features.

[0093] Further, in terms of the generation of the segmentation mask, the task text and the multi-view image are input into a multi-modal segmentation model, the model is guided by the task semantic information to locate and segment the object region related to the task target in the image, and a corresponding binary or multi-value segmentation mask image is output. The segmentation mask can explicitly identify the task attention region in the visual features, reducing background interference. The mask image is input into a convolutional coding network to extract the shape, boundary and spatial distribution features of the target region, and a segmentation feature vector is obtained through linear mapping.

[0094] Further, in terms of the pose information of the actuator, the angle data of the joint encoder of the robot arm, the pose information and speed data of the end effector are read to form a real-time motion state vector of the actuator, and the vector is input into a pose feature coding module. The module preferably adopts a KAN structure or a multi-layer perceptron with nonlinear fitting capability to map the pose information into a low-dimensional pose feature vector, retaining the coupling relationship between joints and the global constraint information of the end pose.

[0095] Further, in terms of the motion frequency information, the motion data of the actuator within a period of time is subjected to Fourier transform or zero-crossing rate analysis to obtain a parameter vector reflecting the motion rhythm and frequency change. The vector is encoded by a KAN module or a fully connected network to generate a frequency feature vector, which is used to provide time dynamic constraints in the action prediction stage.

[0096] In this way, the embodiment generates depth features, optical flow features, segmentation features, pose features and frequency features, which together with the aforementioned trajectory features and key point features constitute the multi-modal input required for task execution, providing a comprehensive environmental perception and action state description for the subsequent multi-modal fusion reasoning model, thereby significantly improving the robustness and generalization ability of the control strategy in multi-task and multi-environment.

[0097] S105, the trajectory features, the key point features, the depth features, the optical flow features, the segmentation features, the pose features and the frequency features are jointly input into a multi-modal fusion reasoning model to generate relative action prediction features and convert them into action control instructions. The robot actuator controls the robot actuator to complete the preset task according to the action control instructions.

[0098] In one specific embodiment of the present application, the trajectory features, key point features, depth features, optical flow features, segmentation features, pose features and frequency features are input into a multi-modal fusion inference model. The model aligns and correlates different modal features, and generates relative action prediction features reflecting the current task state and environmental constraints. The relative action prediction features are used to describe the incremental control amount relative to the action at the previous time, including the joint angle change amount, the end pose change amount and the actuator opening and closing change amount. The relative action prediction features are then converted into action control instructions executable by the robot, and sent to the actuator control module to drive the robot to complete the task execution of the current step according to the control instructions.

[0099] As a possible implementation, see Figure 2 The flowchart of a multi-modal embodied intelligent control method provided by the embodiment of the present application is shown in FIG. 1. The method comprises steps S201-S203, wherein:

[0100] S201, sample operation video data of the robot actuator is obtained, and corresponding text is configured for the execution action corresponding to the sample operation video data.

[0101] S202, the motion pose of the robot actuator is labeled as a preset requirement label and a key point label, and an image feature extraction model corresponding to the multi-view image is trained according to the preset requirement label and the key point label.

[0102] S203, after the image feature extraction model is trained, the image feature extraction model is frozen, and the multi-modal fusion inference model is trained, so that the multi-modal fusion inference model has the ability to output relative action prediction features.

[0103] In a specific implementation, the training process is divided into two stages. For the first stage, the image feature extraction model is mainly trained. This model is used to better predict the motion trajectory and key point information. However, this stage does not depend on any motion information and robot structure, and can train the model based on a large amount of video data related to network mechanical arm operation.

[0104] First, the action performed by the video data is equipped with corresponding text, and the overall motion pose of the mechanical arm is labeled as a model prediction label and a key point label. Then, the image feature extraction model is continuously trained based on the label, so that the model can better predict the action trajectory corresponding to the task action.

[0105] For the second stage freeze corresponds to the trained image feature extraction model, and then based on the multi-modal fusion inference model to predict the need to perform the action features, the feature training using MSE loss function. In the model inference also need to convert the feature into the corresponding mechanical arm each joint angle information, so that the robot can be based on the joint angle change to perform the corresponding task.

[0106] To improve the environmental adaptability and multi-modal information fusion ability of the model, a multi-level data augmentation strategy is used in the training process. Image random mask: random region occlusion is performed on the input camera image, the mask area accounts for 10%-30%(randomly generated rectangular or irregular polygon mask), and it is ensured that the mask area in a single image does not exceed 50% of the key semantic area(such as the end of the mechanical arm, the operation target), forcing the model to reduce the dependence on local image information; image augmentation: including color jittering(brightness adjustment range ±20%, contrast adjustment range 0.8-1.2 times, saturation adjustment range 0.7-1.3 times), arbitrary cropping(cropping image size remains 60%-100% of the original size, and the center of the cropped area is offset from the center of the original image by no more than 20% of the original size).

[0107] These operations not only make the model adapt to different light intensity and color temperature conditions, but also destroy the integrity of the single camera image, prompting the model to fuse the complementary information of the three cameras(such as different target profiles and mechanical arm poses), and improve the global consistency of behavior prediction.

[0108] In the prediction execution phase, the mechanical arm uses a closed-loop execution mechanism of sequence traversal-dynamic inference: first, based on the motion sequence(output by the model) containing joint angular velocity, end movement distance and other parameters at each time step, the frame is traversed and executed, and after completing an action at each time step, the current environment image and mechanical arm state data are collected in real time, and the model is re-input for inference to generate the next motion sequence. This process continues to iterate until the mechanical arm completes the preset action goal(such as grabbing the target object and placing it in the specified location), thereby correcting the cumulative error through dynamic feedback to ensure the execution accuracy of complex actions.

[0109] The multi-modal embodiment of the present disclosure provides a body intelligent control method, which acquires multi-view images captured by cameras at multiple parts of a robot and task text; generates two-dimensional trajectory supervision information and key point supervision information based on the task text and the multi-view images, and converts the two-dimensional trajectory supervision information and the key point supervision information into a visual supervision graph; generates key point features corresponding to the key point supervision information according to the visual supervision graph, and generates corresponding trajectory features and key point features in combination with the multi-view images; determines a depth image corresponding to the multi-view images, optical flow information, a segmentation mask matched with the task text, and posture information and motion frequency information of an executing mechanism, and respectively generates corresponding depth features, optical flow features, segmentation features, posture features and frequency features; inputs the trajectory features, the key point features, the depth features, the optical flow features, the segmentation features, the posture features and the frequency features into a multi-modal fusion inference model, generates relative action prediction features and converts the relative action prediction features into action control instructions; and controls the executing mechanism of the robot to complete a preset task according to the action control instructions. The body intelligent method introduced under the visual-language condition and based on trajectory supervision, key point supervision and multi-modal prior features can more stably learn the mapping relationship from multi-modal perception to fine actions, significantly improves the integrity and robustness of task perception, and has higher execution accuracy.

[0110] Those skilled in the art can understand that, in the above method of the specific implementation, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process, and the specific execution order of each step should be determined according to its function and possible internal logic.

[0111] Based on the same inventive concept, the multi-modal embodiment of the present disclosure also provides a multi-modal body intelligent control device corresponding to the multi-modal body intelligent control method. Since the principle of solving problems by the device in the multi-modal embodiment of the present disclosure is similar to the multi-modal body intelligent control method in the multi-modal embodiment of the present disclosure, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described herein.

[0112] Please refer to Figure 3 , Figure 3 A schematic diagram of a multi-modal body intelligent control device provided by the multi-modal embodiment of the present disclosure is shown in FIG. 3. Figure 3 As shown in FIG. 3, the multi-modal body intelligent control device 300 provided by the multi-modal embodiment of the present disclosure includes:

[0113] The data acquisition module 310 is configured to acquire multi-view images captured by cameras at multiple parts of a robot and task text.

[0114] The trajectory and key point supervision module 320 is configured to generate two-dimensional trajectory supervision information and key point supervision information based on the task text and the multi-view images, and convert the two-dimensional trajectory supervision information and the key point supervision information into a visual supervision graph.

[0115] The first feature extraction module 330 is used to generate key point features corresponding to the key point supervision information based on the visualization supervision map, and to generate corresponding trajectory features and key point features in combination with the multi-view image.

[0116] The second feature extraction module 340 is used to determine the depth image, optical flow information, segmentation mask matching the task text, and pose information and motion frequency information corresponding to the multi-view image, and generate corresponding depth features, optical flow features, segmentation features, pose features and frequency features respectively.

[0117] The motion prediction module 350 is used to input the trajectory features, key point features, depth features, optical flow features, segmentation features, pose features and frequency features into a multimodal fusion inference model to generate relative motion prediction features and convert them into motion control commands.

[0118] The control execution module 360 ​​is used to control the robot's actuator to complete the preset task according to the action control command.

[0119] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0120] This disclosure provides a multimodal embodied intelligent control device that acquires multi-view images taken by cameras at multiple locations on a robot, as well as task text. Based on the task text and the multi-view images, it generates two-dimensional trajectory supervision information and key point supervision information, and converts them into a visual supervision map. According to the visual supervision map, it generates key point features corresponding to the key point supervision information, and combines the multi-view images to generate corresponding trajectory features and key point features. It determines the depth image, optical flow information, and segmentation mask matching the task text corresponding to the multi-view images, as well as the posture information and motion frequency information corresponding to the actuator, and generates corresponding depth features, optical flow features, segmentation features, posture features, and frequency features respectively. It inputs the trajectory features, key point features, depth features, optical flow features, segmentation features, posture features, and frequency features into a multimodal fusion inference model to generate relative motion prediction features and convert them into motion control commands. It controls the robot actuator to complete a preset task according to the motion control commands. Embodied intelligence methods that incorporate trajectory supervision, keypoint supervision, and multimodal prior features under vision-language conditions can more stably learn the mapping relationship from multimodal perception to fine movements, significantly improving the integrity and robustness of task perception and achieving higher execution accuracy.

[0121] Corresponding to Figure 1The multi-modal embodiment intelligent control method in the embodiment of the present disclosure, the present disclosure also provides an electronic device 400, as shown in Figure 4 The electronic device 400 structure schematic diagram provided by the embodiment of the present disclosure, including:

[0122] The processor 41, the memory 42, and the bus 43; the memory 42 is used for storing execution instructions, including the memory 421 and the external memory 422; the memory 421 here is also called the internal memory, used for temporarily storing the operation data in the processor 41 and the data exchanged with the external memory 422 such as a hard disk, the processor 41 exchanges data with the external memory 422 through the memory 421, when the electronic device 400 runs, the processor 41 and the memory 42 communicate through the bus 43, so that the processor 41 executes Figure 1 The steps of the multi-modal embodiment intelligent control method in the embodiment of the present disclosure.

[0123] The embodiment of the present disclosure also provides a computer readable storage medium, the computer readable storage medium has a computer program, the computer program is run by the processor to execute the steps of the multi-modal embodiment intelligent control method described in the above method embodiment. Wherein, the storage medium can be a volatile or non-volatile computer readable storage medium.

[0124] The embodiment of the present disclosure also provides a computer program product, the computer program product includes computer instructions, the computer instructions are executed by the processor to execute the steps of the multi-modal embodiment intelligent control method described in the above method embodiment, for reference can be seen from the above method embodiment, here will not be repeated.

[0125] Among them, the above-mentioned computer program product can be realized by hardware, software or combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, in another optional embodiment, the computer program product is embodied as a software product, such as software development kit (Software Development Kit, SDK) and the like.

[0126] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the apparatus described above can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here. In several embodiments provided in the present disclosure, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are only schematic. For example, the division of the units is only a logical function division, and another division can be made in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interface, and can be electrical, mechanical or other forms.

[0127] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0128] In addition, the functional units in each embodiment of the present disclosure can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0129] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present disclosure essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present disclosure. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0130] Finally, it should be noted that the above-described embodiments are merely specific embodiments of the present disclosure, used to illustrate the technical solutions of the present disclosure, and are not intended to limit the present disclosure. The protection scope of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easy changes to the technical solutions described in the foregoing embodiments, or easily think of changes or equivalent replacements for some of the technical features; and these modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A multimodal embodied intelligent control method, characterized in that, include: Acquire multi-view images taken by cameras at multiple locations on the robot, along with task text; Based on the task text and the multi-view images, two-dimensional trajectory supervision information and key point supervision information are generated and converted into a visual supervision map; Based on the visualization supervision map, key point features corresponding to the key point supervision information are generated, and corresponding trajectory features and key point features are generated by combining the multi-view images. Determine the depth image, optical flow information, and segmentation mask matching the task text corresponding to the multi-view image, as well as the pose information and motion frequency information corresponding to the actuator, and generate the corresponding depth features, optical flow features, segmentation features, pose features, and frequency features respectively. The trajectory features, key point features, depth features, optical flow features, segmentation features, pose features, and frequency features are all input into a multimodal fusion inference model to generate relative motion prediction features and convert them into motion control commands. The robot's actuators are controlled to complete preset tasks according to the motion control instructions.

2. The method according to claim 1, characterized in that, Based on the task text and the multi-view images, two-dimensional trajectory supervision information and key point supervision information are generated, specifically including: The image information from the robot's left arm, right arm, and head camera in the multi-view image is input into the image encoder and linearly projected to obtain the corresponding image visual features. The task text is input into a text encoder to obtain task text features, and multimodal fusion is performed on the image visual features and the task text features to output the two-dimensional trajectory supervision information and the key point supervision information of the text modality. Specifically, the two-dimensional coordinates and execution state of the robot's end effector in a preset viewpoint image coordinate system are extracted, and the trajectory sequence information of the text modality is generated as the two-dimensional trajectory supervision information; The entity words of the manipulated object are parsed from the task text, and the key point positions of the manipulated object and the robot end effector are predicted. The key point coordinates and key point names of the text modality are generated as the key point supervision information.

3. The method according to claim 1, characterized in that, Converting the two-dimensional trajectory supervision information and the key point supervision information into a visual supervision map specifically includes: Based on the two-dimensional trajectory supervision information, a trajectory supervision image is generated, which is based on the robot head camera view and includes the two-dimensional trajectory path and the corresponding actuator state of the robot end effector. Based on the key point supervision information, a key point supervision image is generated, which is based on the view of the robot's head camera and includes the key point positions of the manipulated object and the robot's end effector.

4. The method according to claim 3, characterized in that: For the trajectory supervision image, the starting point, path, and ending point in the two-dimensional trajectory path are encoded with different colors, and the action nodes corresponding to the robot end effector are marked in the two-dimensional trajectory path using a preset pattern, and different states of the actuator are distinguished by color; For the key point supervision image, based on the key point position, the key points corresponding to the manipulated object and the robot end effector are marked with the preset pattern and color-coded.

5. The method according to claim 3, characterized in that, Based on the visualized supervision map, key point features corresponding to the key point supervision information are generated, and corresponding trajectory features and key point features are generated in combination with the multi-view images, specifically including: The key point supervision image is subjected to convolution and linear projection to extract the key point features; The trajectory supervision image is multiplied by a preset first coefficient, and the image information captured by the robot's head camera is multiplied by a preset second coefficient. Then, different levels of transparency are used to fuse the images to generate a global trajectory running image from the robot's head perspective. The trajectory features are generated by performing convolution and linear projection on the global trajectory image.

6. The method according to claim 1, characterized in that, The optical flow feature is generated based on the following steps: For each viewpoint camera image, extract the starting camera image and the ending camera image of the trajectory sequence under that viewpoint, and determine the optical flow information between the starting camera image and the ending camera image under that viewpoint; For the optical flow information under each viewpoint, reprojection is performed to extract projection features. After performing cross-attention operation on the projection features under the robot's left arm and right arm viewpoints and the projection features under the robot's head viewpoint, the resulting features of the cross-attention operation are concatenated to determine the optical flow features.

7. The method according to claim 1, characterized in that, The model is trained based on the following steps: Acquire sample operation video data of the robot actuator, and configure corresponding text for the execution actions corresponding to the sample operation video data; The motion posture of the robot actuator is labeled as a preset requirement label and a key point label, and an image feature extraction model corresponding to the multi-view image is trained based on the preset requirement label and the key point label; After the image feature extraction model is trained, the image feature extraction model is frozen, and the multimodal fusion inference model is trained so that the multimodal fusion inference model has the ability to output relative action prediction features.

8. A multimodal embodied intelligent control device, characterized in that, include: The data acquisition module is used to acquire multi-view images taken by cameras at multiple locations on the robot, as well as task text; The trajectory and key point supervision module is used to generate two-dimensional trajectory supervision information and key point supervision information based on the task text and the multi-view image, and convert them into a visual supervision map; The first feature extraction module is used to generate key point features corresponding to the key point supervision information based on the visualization supervision map, and to generate corresponding trajectory features and key point features in combination with the multi-view image. The second feature extraction module is used to determine the depth image, optical flow information, segmentation mask matching the task text, and pose information and motion frequency information corresponding to the multi-view image, and generate corresponding depth features, optical flow features, segmentation features, pose features and frequency features respectively. The motion prediction module is used to input the trajectory features, key point features, depth features, optical flow features, segmentation features, pose features and frequency features into a multimodal fusion inference model to generate relative motion prediction features and convert them into motion control commands. The control execution module is used to control the robot's actuator to complete the preset task according to the action control instructions.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the multimodal embodied intelligent control method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the multimodal embodied intelligent control method as described in any one of claims 1 to 7.

Images