Multi-modal surgical robot control method and device based on diffusion model
Through a multimodal surgical robot control method based on a diffusion model, combined with video and language information, real-time anomaly detection and dynamic path planning, the limitations of existing surgical robot control methods are overcome, and efficient task completion in complex environments is achieved.
Patent Information
- Application Number
- CN202510552668.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-29
- Publication Date
- 2025-09-26
AI Technical Summary
Existing surgical robot control methods are limited to preset instructions and have a single processing method, which makes it difficult to adapt to the complex and dynamic robotic arm operating environment.
A multimodal surgical robot control method based on a diffusion model is adopted. By acquiring video and language information, a pre-trained robot control large model is used for motion generation and language understanding, combined with real-time anomaly detection and dynamic path planning, to achieve flexibility and adaptability of robot control.
It realizes the flexibility and adaptability of robot control, can efficiently complete tasks in complex environments, improve task completion rate and safety, and has a wide range of applications.
Smart Images
Figure CN120694752A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of medical image processing technology, and in particular to a multimodal surgical robot control method and device based on a diffusion model. Background Art
[0002] Current surgical robots generally perform corresponding operations through a number of pre-built instructions; this form of preset instructions is single and rigid, and the processing method is too limited. Summary of the Invention
[0003] The problem addressed by this application is the limitations of current surgical robot control processing methods.
[0004] To solve the above problems, the first aspect of the present application provides a multimodal surgical robot control method based on a diffusion model, which includes:
[0005] Acquiring multimodal control information, where the multimodal control information includes video information and language information;
[0006] Inputting the multimodal control information into a pre-trained robot control model to obtain robot control information;
[0007] Based on the robot control information, the robot is controlled; the robot control model includes an action generation module and a language understanding module, and the action generation module is obtained based on diffusion model training.
[0008] A second aspect of the present application provides a manufacturing system for a multimodal surgical robot control method based on a diffusion model, comprising:
[0009] An information acquisition module, configured to acquire multimodal control information, wherein the multimodal control information includes video information and language information;
[0010] a control generation module, configured to input the multimodal control information into a pre-trained robot control model to obtain robot control information;
[0011] The robot control module is used to control the robot based on the robot control information; the robot control model includes an action generation module and a language understanding module, and the action generation module is obtained based on the diffusion model training.
[0012] A third aspect of the present application provides an electronic device, comprising: a memory and a processor; the memory may be configured to store a program; the processor is coupled to the memory and is configured to execute the program in the memory, so as to:
[0013] Acquiring multimodal control information, where the multimodal control information includes video information and language information;
[0014] Inputting the multimodal control information into a pre-trained robot control model to obtain robot control information;
[0015] Based on the robot control information, the robot is controlled; the robot control model includes an action generation module and a language understanding module, and the action generation module is obtained based on diffusion model training.
[0016] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the aforementioned diffusion model-based multimodal surgical robot control method.
[0017] In this application, control instructions can be flexibly obtained by obtaining multimodal control information; by presetting a large robot control model, corresponding control information can be generated for various forms of control instructions to achieve control, which has a wide range of applications and greatly increases the processing methods of control instructions. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 is a flow chart of a multimodal surgical robot control method based on a diffusion model according to an embodiment of the present application;
[0019] Figure 2 1 is an architectural diagram of a large model of a multimodal surgical robot control method based on a diffusion model according to an embodiment of the present application;
[0020] Figure 3 1 is an architectural diagram of a multimodal surgical robot control device based on a diffusion model according to an embodiment of the present application;
[0021] Figure 4 2 is a diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0022] To make the above-mentioned objects, features, and advantages of the present application more clearly understood, specific embodiments of the present application are described in detail below with reference to the accompanying drawings. Although the accompanying drawings show exemplary embodiments of the present application, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.
[0023] It should be noted that, unless otherwise specified, the technical or scientific terms used in this application should have the common meanings understood by those skilled in the art in this application.
[0024] The embodiment of the present application provides the above-mentioned multimodal surgical robot control method based on the diffusion model, and the specific scheme of the method is as follows: Figure 1-Figure 2 As shown, the method can be performed by a multimodal surgical robot control device based on a diffusion model, and the multimodal surgical robot control device based on a diffusion model can be integrated into electronic devices such as computers, servers, computers, server clusters, and data centers. Figure 1 As shown, the multimodal surgical robot control method based on the diffusion model includes:
[0025] S101, acquiring multimodal control information, where the multimodal control information includes video information and language information;
[0026] In this application, a high-resolution RGB-D camera (such as Intel RealSense or Kinect) or a LiDAR device is used to capture real-time video streams of the robot's work area. Ensure that the camera covers the robot's work area and minimizes blind spots.
[0027] In this application, a microphone array is used to collect the user's voice commands. A speech recognition model (such as Google ASR or a custom trained model) is applied to convert the speech into text.
[0028] In this application, the video information and the language information are time-synchronized to ensure that the timestamps of the two are consistent. For example, the voice command is associated with the video content of the current frame.
[0029] S102, inputting the multimodal control information into a pre-trained robot control model to obtain robot control information;
[0030] S103, controlling the robot based on the robot control information; the robot control model includes an action generation module and a language understanding module, and the action generation module is obtained based on diffusion model training.
[0031] In this application, a pre-trained large robot control model analyzes multimodal control information to generate robot control information. The language understanding module combines video and language information to analyze user intent. The action generation module generates specific robot control paths based on the diffusion model.
[0032] In this application, control instructions can be flexibly obtained by obtaining multimodal control information; by presetting a large robot control model, corresponding control information can be generated for various forms of control instructions to achieve control, which has a wide range of applications and greatly increases the processing methods of control instructions.
[0033] Furthermore, path planning based on control instructions currently generates each node along the planned path incrementally. This single-path generation approach makes it difficult to meet the needs of the increasingly complex robotic arm operating environment.
[0034] Conventional path planning methods typically generate a single path solution, lacking diversity. This approach can easily lead to the accumulation of incorrect path planning experiences in complex scenarios like obstacle avoidance, resulting in poor overall planning performance.
[0035] For example, in a complex 3D environment, there may be multiple paths to the goal (such as bypassing an obstacle on the left or an obstacle on the right). Traditional methods usually only find one of the paths and ignore other possibilities.
[0036] In this application, by introducing a diffusion model and utilizing the irregular distribution of noise in the diffusion model, a feasible path is randomly generated, thereby solving the problem that the robot arm path planning generation method is single and difficult to meet the increasingly complex robot arm operating environment.
[0037] In one embodiment, the step S103 of controlling the robot based on the robot control information includes:
[0038] Obtain video information during robot control process;
[0039] Based on video information, the robot can detect anomalies;
[0040] In abnormal situations, the robot control information is updated based on the video information and action generation model;
[0041] The robot is controlled based on the updated robot control information.
[0042] In this application, tasks are performed according to robot control information, and real-time video information is used to perform anomaly detection and dynamic adjustment during the execution process.
[0043] In this application, real-time video capture involves continuously capturing a video stream of the robot's work area using an RGB-D camera or LiDAR device. Semantic segmentation updates involve applying a semantic segmentation model to segment video frames in real time to extract dynamic changes in the environment (such as new obstacles or the movement of target objects).
[0044] Anomaly detection: Use optical flow methods (such as the Farne back algorithm) or deep learning methods (such as Flow Net) to detect dynamic changes in the video. Anomalies include: new obstacles entering the robot's working area, the displacement or disappearance of the target object, and the robot deviating from the planned path.
[0045] Dynamic Path Replanning: This module calls the action generation module (based on the diffusion model) to regenerate the path. Inputs include the latest 3D environment model and semantic segmentation embeddings, as well as the current robot state. Output is the updated robot control path. Based on this updated control information, the robot continues to execute the task. Real-time video stream monitoring ensures the robot successfully completes the task.
[0046] In this application, during execution, video information is continuously collected and semantic segmentation is updated. Anomaly detection is performed based on the video information, and the action generation module is called upon to replan the path in case of anomalies. Based on the updated control information, the robot's behavior is dynamically adjusted to ensure successful task completion.
[0047] This application leverages the advantages of multimodal data (video and speech) and combines it with the powerful generative capabilities of diffusion models to achieve intelligent robot control in dynamic environments. Through real-time video monitoring and anomaly detection, the system can quickly adapt to environmental changes, significantly improving task completion and safety.
[0048] In one embodiment, combined Figure 2 As shown, the robot controls the training process of the large model, including:
[0049] Acquiring historical path information and historical multimodal information, wherein the historical multimodal information includes historical video information and historical language information, and the historical path information and the historical video information have a corresponding relationship;
[0050] Based on historical video information and historical path information, the action generation module of the robot control large model is trained to obtain a trained action generation module;
[0051] Based on historical multimodal information and historical path information, the robot control large model is trained to obtain a trained robot control large model.
[0052] In this way, the action generation module is trained first, and then the action generation module is kept unchanged and the entire robot control model is trained, thereby realizing separate training of different modules, improving the training effect, and avoiding interference between modules with different logical angles during the training process.
[0053] This application leverages the diversity of historical data and the complementarity of multimodal information to significantly enhance the intelligence and dynamic adaptability of large-scale robotic control models. By leveraging the action generation module of the diffusion model and the overall framework of joint training, the system is able to efficiently complete tasks in complex dynamic environments.
[0054] In one embodiment, the motion training of the motion generation module of the robot control large model based on the historical video information and the historical path information to obtain the trained motion generation module includes:
[0055] Conduct preliminary training of noise prediction models based on public datasets;
[0056] Grouping historical video information and historical path information to obtain a group training set;
[0057] Performing fine training on the noise prediction model after preliminary training based on the group training set to obtain a finely trained noise prediction model;
[0058] Traverse all group training sets to obtain the corresponding fine-trained noise prediction model.
[0059] In this application, historical video information and historical path information are grouped to construct a group training set for subsequent fine training.
[0060] In this application, historical data is grouped by task type, environment complexity, or robot state. For example, grouping by task type: grasping tasks, placement tasks, obstacle avoidance tasks, etc. Grouping by environment complexity: simple environment (few obstacles), complex environment (many obstacles). Grouping by robot state: different initial positions, different target positions.
[0061] In this application, each set of data is organized into a training set format to obtain several group training sets, each group corresponding to a specific task type or environmental condition.
[0062] In this application, the noise prediction model after preliminary training is refined using the group training set to adapt it to specific tasks or environments. In group training, the same loss function as the preliminary training is used, but more task-related constraints are added.
[0063] In this application, by traversing all group training sets, the noise prediction model is finely trained respectively to obtain multiple models adapted to different tasks or environments.
[0064] In this application, group training and fine training are used to ensure that the noise prediction model can adapt to diverse tasks and environmental requirements, thereby significantly improving the flexibility and robustness of the action generation module.
[0065] In one embodiment, the preliminary training of the noise prediction model based on the public dataset includes:
[0066] Collect the motion trajectory data of the robotic arm in different scenarios and mark the conditional input information;
[0067] Construct an initial noise prediction model;
[0068] The motion trajectory data and conditional input information are gradually denoised to obtain the final noise, and the added noise is recorded;
[0069] Based on the noise prediction model, the final noise and conditional input information are predicted to obtain the predicted noise;
[0070] Calculating an overall loss of a noise prediction model based on the predicted noise and the added noise;
[0071] The noise prediction model is iterated based on the overall loss until the loss converges.
[0072] In this application, the conditional input information may be a 3D environment model. The 3D environment model is semantically segmented to obtain a semantic segmentation embedding. The noise prediction model and the semantic segmentation embedding are used to train the noise prediction model.
[0073] The motion trajectory data and conditional input information are gradually denoised to achieve forward diffusion, obtaining the noise of each diffusion step and the final noise; the noise prediction model is input based on the final noise and semantic segmentation embedding (conditional input information) to obtain the predicted noise.
[0074] In each step of the forward diffusion and backward diffusion, the semantic segmentation embedding remains consistent.
[0075] In one embodiment, inputting the noise prediction model based on the final noise and the semantic segmentation embedding (conditional input information) to obtain predicted noise includes: combining the semantic segmentation embedding and the final noise to obtain a combined feature map;
[0076] The combined feature map is input into the noise prediction model to obtain the predicted noise.
[0077] In one embodiment, performing semantic segmentation on the 3D environment model to obtain a semantic segmentation embedding includes:
[0078] Input the 3D environment model into the pre-trained large recognition model to obtain the obstacle distribution map;
[0079] Encode the obstacle distribution map to obtain semantic segmentation information;
[0080] The semantic segmentation information is flattened to obtain the semantic segmentation embedding.
[0081] In this application, a pre-trained semantic segmentation model is used to perform semantic segmentation on the 3D environment model; an encoder (such as a convolutional neural network (CNN) or a Transformer) is used to extract features from the obstacle distribution map. The encoder converts the obstacle distribution map into a high-dimensional feature vector that represents the location and distribution information of each semantic category. The high-dimensional feature vector is flattened to generate a one-dimensional semantic segmentation embedding. The flattened embedding vector can be directly used as the conditional input of the noise prediction model.
[0082] In this application, semantic segmentation embeddings contain key information about the 3D environment (such as obstacle distribution), providing environmental awareness for the noise prediction model. By flattening, semantic segmentation embeddings can be integrated into the diffusion model input in a compact form, improving the model's efficiency and performance.
[0083] In one embodiment, the step S102 of inputting the multimodal control information into a pre-trained robot control model to obtain robot control information includes:
[0084] Input the video information and language information into the language understanding module of the robot control model to obtain the task execution sequence;
[0085] The task execution sequence and video information are input into the action generation module of the robot control model to obtain robot control information; the robot control information includes multiple execution segment paths, and each execution segment path corresponds one-to-one to an execution segment in the task execution sequence.
[0086] In this application, multimodal control information is parsed through a pre-trained robot control large model: the language understanding module parses language information and video information to generate a task execution sequence; the action generation module generates robot control information based on the task execution sequence and video information.
[0087] In one embodiment, the step of inputting the video information and language information into the language understanding module of the robot control model to obtain the task execution sequence includes:
[0088] Extracting key information from the language information, wherein the key information includes a target object, an action, or a target area;
[0089] Perform semantic segmentation on the video information to obtain multiple target objects and target areas in the video information;
[0090] Extract the position information and posture information of the target object and target area;
[0091] Cross-analyze the target object and target area of the video information with the target object and target area of the language information to obtain a cross-analysis result;
[0092] Based on the cross-analysis results, the task is decomposed into multiple execution segments to form a task execution sequence.
[0093] In this application, the language understanding module generates a task execution sequence: combining video information and language information, analyzing user intentions, and generating a task execution sequence.
[0094] In this application, user voice commands are converted into text after speech recognition. Natural language processing (NLP) technology is used to extract key information. Video frames or point cloud data are collected in real time.
[0095] In this application, a semantic segmentation model (such as DeepLabV3+ and PointNet++) is used to segment the video information and mark multiple target objects and target areas in the video.
[0096] In this application, the position information (such as three-dimensional coordinates) and posture information (such as direction angle) of each target object and target area are extracted.
[0097] In this application, the target object and target area of the language information are matched with the target object and target area in the video information. For example, if the language information mentions "red cup", the specific position and posture of the "red cup" in the video are found.
[0098] In this application, the task is decomposed into multiple execution segments according to the user intention and the environment status.
[0099] In this application, the action generation module generates robot control information (i.e., path planning) based on the task execution sequence and video information. The task execution sequence and video information are input, and a path is generated based on a diffusion model: each execution segment in the task execution sequence is traversed and the corresponding path is generated in sequence; each execution segment path corresponds one-to-one to an execution segment in the task execution sequence. The robot control information is output, including multiple execution segment paths.
[0100] This application leverages the advantages of multimodal data (video and language) and combines it with the powerful generation capabilities of diffusion models to automate the entire process, from user intent analysis to path planning. By combining language understanding and action generation modules, the system can efficiently and accurately complete complex tasks while also possessing strong dynamic adaptability.
[0101] In one embodiment, before semantic segmentation is performed on the 3D environment model and semantic segmentation embedding is obtained, the 3D environment is also updated. The specific process of the update includes:
[0102] Divide the three-dimensional environment model into blocks to obtain independent blocks;
[0103] For each independent block, obtain a first neighboring block and a second neighboring block with different spacings;
[0104] generating a first feature block based on the independent block and the first neighborhood block;
[0105] generating a second feature block based on the independent block and the second neighborhood block;
[0106] Performing feature compression on the first feature block and the second feature block to obtain a compressed block;
[0107] Iterate over all independent blocks and generate an updated 3D environment model based on the resulting compressed blocks.
[0108] In this application, the three-dimensional environment model is divided into blocks, that is, the three-dimensional environment model is divided into corresponding image blocks through a checkerboard grid; wherein, the image block can be at the pixel level (that is, each pixel is an image block) or at other levels, and the specific division shall be based on the actual processing situation.
[0109] In this application, a sliding window or a fixed step size is used to divide the image into blocks of the same size.
[0110] It should be noted that the 3D environment model is a 3D image, and a surface is selected for chessboard division, each grid is a strip grid with a lot of depth (the depth is the depth of the 3D image), and the strip grid is an image block.
[0111] Preferably, in the present application, each image block is 1001000 pixels, so that more feature calculations between local areas can be performed on the basis of ensuring generation accuracy and reducing the amount of calculation.
[0112] In this application, an image block is selected as an independent block. The adjacent image blocks above, below, to the left, and to the right of the independent block are the first neighboring blocks. The image blocks above, below, to the left, and to the right of the independent block, separated by one grid, are the second neighboring blocks. The first and second neighboring blocks have different spacing from the independent block.
[0113] In this application, the neighborhood information of each independent block is extracted to capture the local structure.
[0114] In the present application, the first feature block is generated to generate a local feature representation using an independent block and its first neighborhood block. Specifically, the independent block and the first neighborhood block are processed by a convolution layer and an attention layer to obtain the first feature block.
[0115] In this application, the specific structure and specific parameters of the convolutional layer and the attention layer can be obtained according to the training data or determined according to the actual situation.
[0116] It should be noted that, in this application, there are four first neighborhood blocks and multiple first feature blocks.
[0117] In this application, the independent block and the first neighborhood block are processed by the convolution layer and the attention layer to obtain the first feature block. The specific process is: the independent block and the four neighborhood blocks are spliced together to form a multi-channel input, and the convolution layer is used to extract features from the spliced block; the self-attention mechanism or the channel attention mechanism is used to enhance important features, calculate the attention weight, and weight the convolution layer output to enhance important features; the output of the attention layer is split into multiple feature blocks, each feature block corresponds to the processing results of the independent block and at least one neighborhood block.
[0118] In this application, a second feature block is generated to generate a broader local feature representation using an independent block and its second neighborhood block. The specific generation process is the same as that of the first feature block, except that the parameters of the convolution layer and the attention layer are different.
[0119] In this application, the generated feature blocks are compressed into a more compact representation to reduce the amount of computation and retain key information. Feature compression is performed using pooling operations (such as maximum pooling or average pooling) or fully connected layers.
[0120] In this way, through compression, the first and second feature blocks are compressed into a compressed block, which corresponds to the size and position of the independent block and is used to replace the independent block. All image blocks are replaced by the compressed block to obtain an updated three-dimensional environment model.
[0121] In the present application, each image block of the three-dimensional environment model is traversed in a traversal manner to obtain a corresponding compressed block.
[0122] In this application, for image blocks / independent blocks near edges, their first and second neighboring blocks are incomplete. In this case, they are completed by copying the first and second neighboring blocks at relatively similar positions. For example, if the first neighboring block above the independent block does not exist, the first neighboring block below is copied and used as the block above.
[0123] In this application, the processing accuracy of the adjacent image blocks is greatly improved by completion.
[0124] In this application, the similarity relationship between local regions is captured through an adaptive adjustment module, thereby enhancing feature representation.
[0125] An embodiment of the present application provides a multimodal surgical robot control device based on a diffusion model, which is used to execute the multimodal surgical robot control method based on a diffusion model described above in the present application. The multimodal surgical robot control device based on a diffusion model is described in detail below.
[0126] like Figure 3 As shown, the multimodal surgical robot control device based on the diffusion model includes:
[0127] An information acquisition module 101 is configured to acquire multimodal control information, wherein the multimodal control information includes video information and language information;
[0128] A control generation module 102, which is used to input the multimodal control information into a pre-trained robot control model to obtain robot control information;
[0129] The robot control module 103 is used to control the robot based on the robot control information; the robot control model includes an action generation module and a language understanding module, and the action generation module is obtained based on diffusion model training.
[0130] In one embodiment, the robot control module 103 is further configured to:
[0131] Acquire video information during the robot control process; perform abnormality detection on the robot based on the video information; in abnormal situations, update the robot control information based on the video information and the action generation model; and control the robot based on the updated robot control information.
[0132] In one embodiment, the control generation module 102 is further configured to:
[0133] Historical path information and historical multimodal information are obtained, wherein the historical multimodal information includes historical video information and historical language information, and the historical path information and the historical video information have a corresponding relationship; based on the historical video information and the historical path information, an action generation module of the robot control large model is trained to obtain a trained action generation module; based on the historical multimodal information and the historical path information, the robot control large model is trained to obtain a trained robot control large model.
[0134] In one embodiment, the control generation module 102 is further configured to:
[0135] The noise prediction model is preliminarily trained based on the public data set; the historical video information and historical path information are grouped to obtain a group training set; the noise prediction model after the preliminary training is fine-trained based on the group training set to obtain a fine-trained noise prediction model; all group training sets are traversed to obtain the corresponding fine-trained noise prediction model.
[0136] In one embodiment, the control generation module 102 is further configured to:
[0137] The motion trajectory data of the robot arm in different scenarios are collected and the conditional input information is annotated; an initial noise prediction model is constructed; the motion trajectory data and the conditional input information are gradually denoised to obtain the final noise, and the added noise is recorded; based on the noise prediction model, the final noise and the conditional input information are subjected to noise prediction to obtain the predicted noise; based on the predicted noise and the added noise, the overall loss of the noise prediction model is calculated; and based on the overall loss, the noise prediction model is iterated until the loss converges.
[0138] In one embodiment, the control generation module 102 is further configured to:
[0139] The video information and language information are input into the language understanding module of the robot control model to obtain the task execution sequence; the task execution sequence and video information are input into the action generation module of the robot control model to obtain the robot control information; the robot control information includes multiple execution segment paths, and each execution segment path corresponds one-to-one to an execution segment in the task execution sequence.
[0140] In one embodiment, the control generation module 102 is further configured to:
[0141] Extract key information from language information, wherein the key information includes a target object, action, or target area; perform semantic segmentation on video information to obtain multiple target objects and target areas in the video information; extract position information and posture information of the target objects and target areas; cross-analyze the target objects and target areas of the video information with the target objects and target areas of the language information to obtain a cross-analysis result; and based on the cross-analysis result, decompose the task into multiple execution segments to form a task execution sequence.
[0142] The multimodal surgical robot control device based on the diffusion model provided in the above-mentioned embodiment of the present application has a corresponding relationship with the multimodal surgical robot control method based on the diffusion model provided in the embodiment of the present application. Therefore, the specific content in the system has a corresponding relationship with the multimodal surgical robot control method based on the diffusion model. The specific content can refer to the records in the multimodal surgical robot control method based on the diffusion model, and will not be repeated in this application.
[0143] The multimodal surgical robot control device based on the diffusion model provided in the above-mentioned embodiments of the present application and the multimodal surgical robot control method based on the diffusion model provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.
[0144] The above describes the internal functions and structure of the multimodal surgical robot control device based on the diffusion model, such as Figure 4As shown, in practice, the multimodal surgical robot control device based on the diffusion model can be implemented as an electronic device, including: a memory 301 and a processor 303.
[0145] The memory 301 may be configured to store programs.
[0146] In addition, the memory 301 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.
[0147] The memory 301 can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The processor 303 is coupled to the memory 301 and is used to execute the program in the memory 301 to:
[0148] Acquiring multimodal control information, where the multimodal control information includes video information and language information;
[0149] Inputting the multimodal control information into a pre-trained robot control model to obtain robot control information;
[0150] Based on the robot control information, the robot is controlled; the robot control model includes an action generation module and a language understanding module, and the action generation module is obtained based on diffusion model training.
[0151] In one embodiment, the processor 303 is further configured to:
[0152] Acquire video information during the robot control process; perform abnormality detection on the robot based on the video information; in abnormal situations, update the robot control information based on the video information and the action generation model; and control the robot based on the updated robot control information.
[0153] In one embodiment, the processor 303 is further configured to:
[0154] Historical path information and historical multimodal information are obtained, wherein the historical multimodal information includes historical video information and historical language information, and the historical path information and the historical video information have a corresponding relationship; based on the historical video information and the historical path information, an action generation module of the robot control large model is trained to obtain a trained action generation module; based on the historical multimodal information and the historical path information, the robot control large model is trained to obtain a trained robot control large model.
[0155] In one embodiment, the processor 303 is further configured to:
[0156] The noise prediction model is preliminarily trained based on the public data set; the historical video information and historical path information are grouped to obtain a group training set; the noise prediction model after the preliminary training is fine-trained based on the group training set to obtain a fine-trained noise prediction model; all group training sets are traversed to obtain the corresponding fine-trained noise prediction model.
[0157] In one embodiment, the processor 303 is further configured to:
[0158] The motion trajectory data of the robot arm in different scenarios are collected and the conditional input information is annotated; an initial noise prediction model is constructed; the motion trajectory data and the conditional input information are gradually denoised to obtain the final noise, and the added noise is recorded; based on the noise prediction model, the final noise and the conditional input information are subjected to noise prediction to obtain the predicted noise; based on the predicted noise and the added noise, the overall loss of the noise prediction model is calculated; and based on the overall loss, the noise prediction model is iterated until the loss converges.
[0159] In one embodiment, the processor 303 is further configured to:
[0160] The video information and language information are input into the language understanding module of the robot control model to obtain the task execution sequence; the task execution sequence and video information are input into the action generation module of the robot control model to obtain the robot control information; the robot control information includes multiple execution segment paths, and each execution segment path corresponds one-to-one to an execution segment in the task execution sequence.
[0161] In one embodiment, the processor 303 is further configured to:
[0162] Extract key information from language information, wherein the key information includes a target object, action, or target area; perform semantic segmentation on video information to obtain multiple target objects and target areas in the video information; extract position information and posture information of the target objects and target areas; cross-analyze the target objects and target areas of the video information with the target objects and target areas of the language information to obtain a cross-analysis result; and based on the cross-analysis result, decompose the task into multiple execution segments to form a task execution sequence.
[0163] In this application, Figure 4 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 4 Components shown.
[0164] The electronic device provided in this embodiment is based on the same inventive concept as the multimodal surgical robot control method based on the diffusion model provided in the embodiment of the present application, and has the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.
[0165] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage, CDROM, optical storage, etc.) that contain computer-usable program code.
[0166] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.
[0167] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0168] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-permanent storage in a computer-readable medium, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory. Memory is an example of a computer-readable medium.
[0169] The present application also provides a computer-readable storage medium corresponding to the multimodal surgical robot control method based on the diffusion model provided in the aforementioned embodiment, on which a computer program (i.e., a program product) is stored. When the computer program is run by the processor, it will execute the interactive image analysis auxiliary method for 3D aerial imaging provided in any of the aforementioned embodiments.
[0170] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CDROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.
[0171] The computer-readable storage medium provided in the above-mentioned embodiment of the present application and the interactive image analysis auxiliary method for 3D aerial imaging provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.
[0172] It should be noted that, in the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.
[0173] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.
[0174] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.
Claims
1. A multimodal surgical robot control method based on a diffusion model, characterized in that: include: Acquiring multimodal control information, where the multimodal control information includes video information and language information; Inputting the multimodal control information into a pre-trained robot control model to obtain robot control information; Based on the robot control information, the robot is controlled; the robot control model includes an action generation module and a language understanding module, and the action generation module is obtained based on diffusion model training.
2. The multimodal surgical robot control method based on the diffusion model according to claim 1, characterized in that: The controlling of the robot based on the robot control information includes: Obtain video information during robot control process; Based on video information, the robot can detect anomalies; In abnormal situations, the robot control information is updated based on the video information and action generation model; The robot is controlled based on the updated robot control information.
3. The multimodal surgical robot control method based on a diffusion model according to claim 1, wherein the training process of the robot control large model comprises: Acquiring historical path information and historical multimodal information, wherein the historical multimodal information includes historical video information and historical language information, and the historical path information and the historical video information have a corresponding relationship; Based on historical video information and historical path information, the action generation module of the robot control large model is trained to obtain a trained action generation module; Based on historical multimodal information and historical path information, the robot control large model is trained to obtain a trained robot control large model.
4. The multimodal surgical robot control method based on a diffusion model according to claim 3, wherein the action generation module of the robot control model is trained based on historical video information and historical path information to obtain the trained action generation module, comprising: Conduct preliminary training of noise prediction models based on public datasets; Grouping historical video information and historical path information to obtain a group training set; Performing fine training on the noise prediction model after preliminary training based on the group training set to obtain a finely trained noise prediction model; Traverse all group training sets to obtain the corresponding fine-trained noise prediction model.
5. The multimodal surgical robot control method based on a diffusion model according to claim 4, wherein the preliminary training of the noise prediction model based on a public dataset comprises: Collect the motion trajectory data of the robotic arm in different scenarios and mark the conditional input information; Construct an initial noise prediction model; The motion trajectory data and conditional input information are gradually denoised to obtain the final noise, and the added noise is recorded; Based on the noise prediction model, the final noise and conditional input information are predicted to obtain the predicted noise; Calculating an overall loss of a noise prediction model based on the predicted noise and the added noise; The noise prediction model is iterated based on the overall loss until the loss converges.
6. The multimodal surgical robot control method based on a diffusion model according to any one of claims 1 to 5, wherein inputting the multimodal control information into a pre-trained robot control large model to obtain the robot control information comprises: Input the video information and language information into the language understanding module of the robot control model to obtain the task execution sequence; The task execution sequence and video information are input into the action generation module of the robot control model to obtain robot control information; the robot control information includes multiple execution segment paths, and each execution segment path corresponds one-to-one to an execution segment in the task execution sequence.
7. The multimodal surgical robot control method based on a diffusion model according to claim 6, wherein the step of inputting video information and language information into a language understanding module of a large robot control model to obtain a task execution sequence comprises: Extracting key information from the language information, wherein the key information includes a target object, an action, or a target area; Perform semantic segmentation on the video information to obtain multiple target objects and target areas in the video information; Extract the position information and posture information of the target object and target area; Cross-analyze the target object and target area of the video information with the target object and target area of the language information to obtain a cross-analysis result; Based on the cross-analysis results, the task is decomposed into multiple execution segments to form a task execution sequence.
8. A multimodal surgical robot control device based on a diffusion model, characterized in that: include: An information acquisition module, configured to acquire multimodal control information, wherein the multimodal control information includes video information and language information; a control generation module, configured to input the multimodal control information into a pre-trained robot control model to obtain robot control information; The robot control module is used to control the robot based on the robot control information; the robot control model includes an action generation module and a language understanding module, and the action generation module is obtained based on the diffusion model training.
9. An electronic device, characterized in that: include: memory and processor; The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program to: Acquiring multimodal control information, where the multimodal control information includes video information and language information; Inputting the multimodal control information into a pre-trained robot control model to obtain robot control information; Based on the robot control information, the robot is controlled; the robot control model includes an action generation module and a language understanding module, and the action generation module is obtained based on diffusion model training.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by the processor to implement the multimodal surgical robot control method based on the diffusion model as described in any one of claim 17.