Multi-mechanical-arm control method and device based on body intelligence

Abstract

The invention provides a multi-mechanical-arm control method and device based on body intelligence. The method comprises the steps that historical video streams of operations of the same type are obtained; analyzing the historical video stream to obtain historical operation information; acquiring to-be-operated information of the current object; and generating operation planning and execution information of the current object based on the historical operation information, and controlling the multiple mechanical arms to execute the execution information. According to the method and the device, the specific information of the historical operation process is obtained by analyzing the historical data, so that the operation of the current object is planned based on the specific information to control execution of the multiple mechanical arms, omission caused by pure manual experience summarization is avoided, operation optimization based on the historical data is realized, and the operation optimization efficiency is improved. And complete control over the whole multiple mechanical arms is achieved.

Description

Technical Field

[0001] The present application relates to the field of medical image processing technology, and more specifically, to a multi-robot arm control method and device based on embodied intelligence. Background Art

[0002] Surgical robot-assisted orthopedic surgery is an important application of orthopedic surgical robots, especially multi-robotic arm surgical robots, which complete complex orthopedic surgeries through preoperative planning, intraoperative image acquisition and actual execution by robotic arms.

[0003] However, the specific surgical procedures and surgical planning of multi-robotic-arm surgical robots need to follow principles, constraints or optimizations. Relying solely on manual summary of historical experience will lead to omissions and make it impossible to complete the planning and control of the entire multi-robotic-arm system. Summary of the invention

[0004] The problem to be solved by the present application is that the historical experience optimization of the control of multiple robotic arms is currently only performed manually, which is prone to omissions.

[0005] To solve the above problems, the first aspect of the present application provides a multi-robot arm control method based on embodied intelligence, which includes:

[0006] Get historical video streams of the same type of surgery;

[0007] Parsing the historical video stream to obtain historical surgery information;

[0008] Get the pending surgery information of the current object;

[0009] Generate surgical planning and execution information for the current object based on historical surgical information, and control the multi-robot arm to execute the execution information.

[0010] The second aspect of the present application provides a manufacturing system of a multi-robot arm control method based on embodied intelligence, which comprises:

[0011] A video acquisition module, which is used to acquire historical video streams of the same type of surgery;

[0012] A video parsing module, which is used to parse the historical video stream to obtain historical surgery information;

[0013] An information acquisition module, which is used to obtain the pending surgery information of the current object;

[0014] The planning and execution module is used to generate the surgical planning and execution information of the current object based on the historical surgical information, and control the multi-manipulator to execute the execution information.

[0015] The third aspect of the present application provides an electronic device, comprising: a memory and a processor; the memory may be configured to store a program, and the processor is coupled to the memory and is used to execute the program in the memory, so as to:

[0016] Get historical video streams of the same type of surgery;

[0017] Parsing the historical video stream to obtain historical surgery information;

[0018] Get the pending surgery information of the current object;

[0019] Generate surgical planning and execution information for the current object based on historical surgical information, and control the multi-robot arm to execute the execution information.

[0020] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the aforementioned multi-robotic arm control method based on embodied intelligence.

[0021] In the present application, by analyzing historical data, specific information about historical surgical processes is obtained, and the surgery of the current object is planned based on the specific information to control the execution of multiple robotic arms. In this way, omissions caused by simple manual experience summary are avoided, surgical optimization based on historical data is achieved, and complete control of the entire multi-robotic arm is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of a multi-robot arm control method based on embodied intelligence according to an embodiment of the present application;

[0023] Figure 2 It is an architecture diagram of image parsing of a multi-robot control method based on embodied intelligence according to an embodiment of the present application;

[0024] Figure 3 is an architecture diagram of a multi-robot arm control device based on embodied intelligence according to an embodiment of the present application;

[0025] Figure 4 is an architectural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below in conjunction with the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0027] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0028] The embodiment of the present application provides the above-mentioned multi-robot arm control method based on embodied intelligence, and the specific scheme of the method is as follows: Figure 1-Figure 2 As shown, the method can be performed by a multi-robot control device based on embodied intelligence, and the multi-robot control device based on embodied intelligence can be integrated into electronic devices such as computers, servers, computers, server clusters, and data centers. Figure 1 As shown, the multi-robot arm control method based on embodied intelligence includes:

[0029] S101, obtaining a historical video stream of the same type of surgery;

[0030] In this application, ensure that the type of surgery or procedure corresponding to the selected video is consistent with or highly similar to the type of surgery to be performed by the current subject;

[0031] S102, parsing the historical video stream to obtain historical surgery information;

[0032] S103, obtaining the surgery information of the current subject;

[0033] S104, generating surgical planning and execution information of the current object based on historical surgical information, and controlling the multi-robot arm to execute the execution information.

[0034] In the present application, by analyzing historical data, specific information about historical surgical processes is obtained, and the surgery of the current object is planned based on the specific information to control the execution of multiple robotic arms. In this way, omissions caused by simple manual experience summary are avoided, surgical optimization based on historical data is achieved, and complete control of the entire multi-robotic arm is realized.

[0035] In a specific implementation, the step S104 of generating the surgical planning and execution information of the current subject based on the historical surgical information includes:

[0036] Processing the historical surgical information based on inverse reinforcement learning to obtain a historical reward function;

[0037] Construct multiple constraints;

[0038] Based on multi-objective optimization, the execution strategy of multiple robotic arms is trained through reinforcement learning;

[0039] After the training is completed, the execution information with the highest cumulative reward is selected and the multi-manipulator is controlled to execute the execution information.

[0040] It should be noted that, in the case of directly acquiring the execution information, the surgical plan will also be used as the corresponding content (corresponding hidden content) in the execution information.

[0041] Among them, based on multi-objective optimization, the execution strategy of multiple robotic arms is trained through reinforcement learning, including:

[0042] Obtain operating environment data and planning data to build a simulation space;

[0043] Based on the inverse reinforcement learning process and the operating environment data and planning data, a state space and an action space are constructed; the action space includes a plurality of joint axes, each of which has a forward rotation action and a reverse rotation action;

[0044] Build a deep Q network model;

[0045] The obtained reward function is used to first train the deep Q network model based on deep Q network learning;

[0046] Taking the difference constraint as the reward function, the deep Q network model is trained for the second time based on deep Q network learning;

[0047] The first training and the second training are performed alternately until a preset condition is met.

[0048] The second training of the deep Q network model based on deep Q network learning using the difference constraint as the reward function includes:

[0049] Construct a reward function based on the constraints of diversity;

[0050] Constructing an experience replay pool, wherein the experience replay pool includes the action, state, reward, and next state of the agent;

[0051] Sampling from the experience replay pool and calculating the Q value based on the deep Q network model;

[0052] By minimizing the loss function, the weights of the deep Q network model are updated until the training is completed.

[0053] The calculation process of the obstacle constraint is as follows:

[0054] Get the robot arm posture data in the current state;

[0055] Get the current state of the environment data;

[0056] Determine the minimum distance between the robot arm and environmental obstacles based on the robot arm posture data and environmental data;

[0057] When the minimum distance between the robot arm and the environmental obstacle is greater than a preset distance, it is determined that the obstacle constraint is satisfied.

[0058] Wherein, determining the minimum distance between the robotic arm and the environmental obstacle according to the robotic arm posture data and the environmental data includes:

[0059] Performing equivalent processing on the robot arm posture data, the robot arm is equivalent to a plurality of line segments connected in sequence and a preset thickness;

[0060] Perform equivalent processing on the environmental data, and equate the obstacles in the environment to spheres and cylinders that wrap the obstacles;

[0061] Calculate the distance between each line segment of the robot arm and the center of the sphere and the axis segment of the cylinder;

[0062] The minimum distance is determined according to the calculated distance, the preset thickness, the radius of the sphere, and the radius of the cylinder.

[0063] In a specific implementation, the historical surgery information is processed based on inverse reinforcement learning to obtain a historical reward function, including:

[0064] Based on historical surgical information, determine historical trajectory data of multiple robotic arms;

[0065] Preprocessing the historical trajectory data;

[0066] Based on the inverse reinforcement learning algorithm, inferring the corresponding reward function from the preprocessed historical trajectory data;

[0067] The reward function is verified based on a reinforcement learning algorithm, and a final reward function is determined after the verification passes.

[0068] In one implementation, the robotic arm is a six-axis robotic arm, and the historical trajectory data is angle change data of six motors corresponding to the six-axis robotic arm over time.

[0069] In one embodiment, the inference of the corresponding reward function from the preprocessed historical trajectory data based on the inverse reinforcement learning algorithm includes:

[0070] Build simulation space based on operating environment data;

[0071] Construct state space based on operating environment data and historical trajectory data;

[0072] Setting an action space, wherein the action space includes a plurality of joint axes, each joint axis having a forward rotation action and a reverse rotation action;

[0073] Based on the state space and action space, a complete running trajectory is constructed;

[0074] Construct a reward function model;

[0075] Model the maximum entropy of trajectory probability to obtain the maximum entropy model;

[0076] By maximizing the log-likelihood of the expert trajectory, the parameters of the reward function model are iteratively optimized to obtain the optimal reward function parameters.

[0077] In a specific implementation, the constraint conditions include obstacle constraints, and / or robot arm collision constraints, and / or safety boundary constraints.

[0078] The safety boundary constraint may be: preventing the tool from deviating from the path during actual execution.

[0079] Among them, the robot arm collision constraint can be: to avoid the robot arm colliding with other robot arms during actual execution. All robot arm links, end effectors, and potential tool models can be added to the same collision detection system, and the shapes between multiple arms can be spatially crossed to determine whether there is a collision.

[0080] Among them, obstacle constraints refer to the need to ensure that the robot arm (including its end effector, tools, etc.) does not collide with external obstacles in the surgical environment or working environment during path planning and actual execution. External obstacles include both fixed obstacles (such as operating tables, operating tables, walls, instruments and equipment, etc.) and dynamic obstacles (other surgical auxiliary equipment, personnel, moving parts of the patient's body, etc.).

[0081] In the path planning stage, a general collision detection library (such as FCL, Bullet, OpenRAVE, etc.) is used to perform collision checks on the links and end effectors of the robot arm to ensure that they do not intersect with external obstacles on the motion trajectory.

[0082] In a specific implementation, the step S102 of parsing the historical video stream to obtain historical surgery information includes:

[0083] Obtaining process information of the same type of surgery;

[0084] Construct process conversion relationships at different levels and time granularities;

[0085] Decomposing the historical video stream by spatial dimension and temporal dimension to identify each process in the historical video stream;

[0086] Identify the end of the actuator in the process and determine the execution data of the surgical instrument;

[0087] Based on the identified procedure and the execution data, historical surgical information is determined.

[0088] The surgical procedure can be defined at several levels according to different time granularities. From coarse time granularity to fine time granularity, there are surgical stages, steps, activities, and actions.

[0089] Among them, at the coarse time granularity level, such as the surgical stage, each category contains a broader description of the surgical process, so the total number of categories is small and the conversion relationship between different categories is simpler. At the fine time granularity level, such as the surgical step, each category describes a step in the surgical process more specifically, so it contains more precise information, but the total number of categories and surgical process conversion increase the complexity of workflow recognition based on spatiotemporal fusion of key content parsing of surgical scenes. In addition, there is a corresponding relationship between the definitions of different levels of surgery, that is, different surgical steps may belong to the same surgical stage.

[0090] In order to better identify surgical steps at each level during the video parsing stage, it is necessary to predefine the conversion order or possible transfer paths of these steps (such as a directed graph or state machine of the surgical process).

[0091] Among them, time dimension analysis divides the surgical video into timelines or extracts key frames: for example, extracting one frame every second or every few seconds, or detecting the moment of sudden action change / scene switching in the video. Combined with the previously established surgical process conversion relationship, the macro steps may correspond to obvious operation features.

[0092] Among them, spatial dimension analysis detects and tracks surgical instruments, doctors’ hands, tissue and organ positions, etc. in each key frame or continuous frame; identifies instrument shapes and IDs (electric knife, forceps, hooks, etc.), and obtains their positions or postures in the picture;

[0093] In this application, a target detection algorithm (such as Faster-RCNN, YOLO, or a Transformer-based detection model) is used to identify the end of the surgical instrument appearing in the video screen;

[0094] In the present application, the pixel coordinates and bounding box of the end of the instrument are continuously tracked; tracking in the time dimension can generate trajectory data (a series of timestamps + spatial coordinates + posture).

[0095] In the present application, the captured movement information of the instrument end is linked to the stage of the surgical process to obtain the surgical information of the surgical instrument.

[0096] In a specific implementation, decomposing the historical video stream by spatial dimension and temporal dimension to identify each process in the historical video stream includes:

[0097] Decomposing historical video streams into time-sequentially arranged image frames;

[0098] Inputting the continuous image frames into the backbone network, extracting spatial dimension features, and obtaining a spatial sequence composed of continuous spatial features;

[0099] Input the spatial sequence into the jump extraction branch to obtain the process prediction feature;

[0100] Input the spatial sequence into the contrast branch to obtain contrast features;

[0101] According to the spatiotemporal fusion features and contrast features, the process corresponding to the image frame is determined.

[0102] In the present application, the input surgical video is split into several image frames in chronological order, and can also be segmented or key frame sequences are taken to reduce the amount of data.

[0103] Combination Figure 2 As shown, the backbone network adopts a ResNeSt structure consisting of four distraction modules to preliminarily extract visual spatial features in the surgical screen, so as to extract richer global and local features in the surgical scene.

[0104] After the global pooling operation (GAP), a fixed-length sequence is formed by continuous spatial features within a period of time, which serves as the input of the skip extraction branch and the comparison branch.

[0105] In this application, there are two types of blurred frames in the surgical workflow recognition task: samples of different categories with similar spatial features and samples of the same category with large differences in spatial features. The comparison branch combats the problem of spatiotemporal inconsistency by intentionally expanding and contracting the distance between the two types of blurred frames in the feature space.

[0106] The main idea of ​​the entire comparison branch is to instantly strengthen and correct the expression of all confused frames in the feature space, so as to finally search for an implicit space with strong representation ability.

[0107] The cosine distance is used to measure the distance between positive and negative samples and fixed samples, and it is added to the overall loss function of the network, so that the entire network is forced to learn a feature expression and find the most critical information to distinguish samples of different categories.

[0108] In a specific embodiment, combining Figure 2 As shown, the continuous image frames are input into the backbone network, the spatial dimension features are extracted, and a spatial sequence composed of continuous spatial features is obtained, including:

[0109] Perform convolution pooling on the continuous image frames to obtain pooling features;

[0110] Input the pooled features into the feature extraction module to obtain the extracted visual space features;

[0111] The visual spatial features are globally pooled to obtain a spatial sequence consisting of continuous spatial features.

[0112] In this application, ResNeSt is used as the backbone network to effectively extract richer visual features in each frame of the video.

[0113] Specifically, each frame of the continuous image sequence in the input sequence first passes through a standard convolutional layer to obtain the feature vector of the preset channel, and then passes through a maximum pooling operation to provide input information for the subsequent ResNeSt module.

[0114] In this application, four distraction residual modules are applied as spatial feature encoders to extract more features of surgical scenes from video frames.

[0115] In this application, each distraction residual module consists of k group convolutions, where the input is divided into r distraction channels in each group and the distraction operation is performed. After the results of the k groups are connected, the number of channels is changed back to the same size as the input through 1×1 convolution.

[0116] In this application, a global average pooling (GAP) layer is used to output a multi-dimensional feature vector to characterize the spatial features contained in each video frame image initially extracted.

[0117] In a specific implementation, the step of inputting the spatial sequence into the jump extraction branch to obtain the process prediction feature includes:

[0118] Input the spatial sequence into a single-layer LSTM network to obtain the motion information between consecutive frames;

[0119] Map the motion information to obtain the first-level process prediction;

[0120] Analyze the motion information to obtain the second-level process prediction;

[0121] Make corrections based on the first and second levels and determine the corresponding process.

[0122] In this application, since frames from different steps may have similar appearances, relying on the spatial features of a single frame image for workflow recognition usually leads to limited recognition capabilities.

[0123] In order to aggregate temporal features, LSTM is used to process the spatial features of a frame sequence generated from the backbone. A single-layer LSTM is used to capture the temporal correlation information between the input video frame sequences, thereby inferring the surgical workflow step to which the current input frame sequence belongs.

[0124] In this application, the spatial sequence is input into a single-layer LSTM to capture the dependencies between frames in the time dimension (such as instrument movement, camera switching, surgical action changes, etc.). The hidden state of LSTM can encode the time sequence information from the current frame to the past frame to obtain motion information or time sequence features.

[0125] The jump extraction branch introduces a single-layer LSTM network to extract the motion information between consecutive frames to obtain spatiotemporal fusion features. And through a Step-Phase joint loss function, the prediction results at these two levels are mutually corrected to achieve surgical workflow recognition at two levels.

[0126] First, the surgical workflow stages are defined along the chronological order of the entire surgical process, so the state transition rules between them are usually fixed and regular, which makes workflow identification at the surgical stage level relatively easy to achieve. At this point, direct analysis can obtain the second-level process prediction.

[0127] Secondly, the state transition relationship at the surgical step level is relatively complex, making it difficult to capture the temporal relationship. Therefore, introducing stage-level recognition is more reliable than directly identifying surgical steps.

[0128] In this application, a mapping function from surgical steps to stages is introduced to achieve surgical workflow prediction at the stage level (first-level process prediction).

[0129] The prediction results of the two levels are combined to correct each other. When there is a contradiction between the second level and the first level, correction can be made through confidence judgment or based on the existing process priors.

[0130] An embodiment of the present application provides a multi-robotic arm control device based on embodied intelligence, which is used to execute the multi-robotic arm control method based on embodied intelligence described in the above content of the present application. The multi-robotic arm control device based on embodied intelligence is described in detail below.

[0131] like Figure 3 As shown, the multi-manipulator control device based on embodied intelligence includes:

[0132] A video acquisition module 101 is used to acquire a historical video stream of the same type of surgery;

[0133] A video parsing module 102, which is used to parse the historical video stream to obtain historical surgery information;

[0134] An information acquisition module 103 is used to acquire the pending surgery information of the current object;

[0135] The planning and execution module 104 is used to generate the surgery planning and execution information of the current object based on the historical surgery information, and control the multi-manipulator to execute the execution information.

[0136] In one embodiment, the planning execution module 104 is further configured to:

[0137] The historical surgical information is processed based on inverse reinforcement learning to obtain a historical reward function; multiple constraints are constructed; based on multi-objective optimization, the execution strategy of the multi-manipulator is trained through reinforcement learning; after the training is completed, the execution information with the highest cumulative reward is selected, and the multi-manipulator is controlled to execute the execution information.

[0138] In one embodiment, the constraint conditions include obstacle constraints, and / or robot arm collision constraints, and / or safety boundary constraints.

[0139] In one implementation, the video parsing module 102 is further configured to:

[0140] Obtain process information of the same type of surgery; construct process conversion relationships at different levels and time granularities; decompose the historical video stream through spatial dimensions and temporal dimensions, and identify each process in the historical video stream; identify the actuator end in the process and determine the execution data of the surgical instrument; determine the historical surgery information based on the identified process and the execution data.

[0141] In one implementation, the video parsing module 102 is further configured to:

[0142] Decompose the historical video stream into image frames arranged in time sequence; input the continuous image frames into the backbone network, extract the spatial dimension features, and obtain a spatial sequence composed of continuous spatial features; input the spatial sequence into the jump extraction branch to obtain the process prediction feature; input the spatial sequence into the comparison branch to obtain the comparison feature; determine the process corresponding to the image frame based on the spatiotemporal fusion features and the comparison features.

[0143] In one implementation, the video parsing module 102 is further configured to:

[0144] The continuous image frames are convolutionally pooled to obtain pooled features; the pooled features are input into a feature extraction module to obtain extracted visual space features; the visual space features are globally pooled to obtain a spatial sequence composed of continuous space features.

[0145] In one implementation, the video parsing module 102 is further configured to:

[0146] The spatial sequence is input into a single-layer LSTM network to obtain the motion information between consecutive frames; the motion information is mapped to obtain the first-level process prediction; the motion information is analyzed to obtain the second-level process prediction; corrections are made based on the first and second levels to determine the corresponding process.

[0147] The multi-robot arm control device based on embodied intelligence provided in the above-mentioned embodiment of the present application has a corresponding relationship with the multi-robot arm control method based on embodied intelligence provided in the embodiment of the present application. Therefore, the specific content in the system has a corresponding relationship with the multi-robot arm control method based on embodied intelligence. The specific content can refer to the records in the multi-robot arm control method based on embodied intelligence, and will not be repeated in this application.

[0148] The multi-robot arm control device based on embodied intelligence provided in the above-mentioned embodiments of the present application and the multi-robot arm control method based on embodied intelligence provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0149] The above describes the internal functions and structure of the multi-manipulator control device based on embodied intelligence, such as Figure 4 As shown, in practice, the multi-robot arm control device based on embodied intelligence can be implemented as an electronic device, including: a memory 301 and a processor 303.

[0150] The memory 301 may be configured to store programs.

[0151] In addition, the memory 301 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0152] The memory 301 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The processor 303, coupled to the memory 301, is used to execute the program in the memory 301, so as to:

[0153] Get historical video streams of the same type of surgery;

[0154] Parsing the historical video stream to obtain historical surgery information;

[0155] Get the pending surgery information of the current object;

[0156] Generate surgical planning and execution information for the current object based on historical surgical information, and control the multi-robot arm to execute the execution information.

[0157] In one implementation, the processor 303 is further configured to:

[0158] The historical surgical information is processed based on inverse reinforcement learning to obtain a historical reward function; multiple constraints are constructed; based on multi-objective optimization, the execution strategy of the multi-manipulator is trained through reinforcement learning; after the training is completed, the execution information with the highest cumulative reward is selected, and the multi-manipulator is controlled to execute the execution information.

[0159] In one embodiment, the constraint conditions include obstacle constraints, and / or robot arm collision constraints, and / or safety boundary constraints.

[0160] In one implementation, the processor 303 is further configured to:

[0161] Obtain process information of the same type of surgery; construct process conversion relationships at different levels and time granularities; decompose the historical video stream through spatial dimensions and temporal dimensions, and identify each process in the historical video stream; identify the actuator end in the process and determine the execution data of the surgical instrument; determine the historical surgery information based on the identified process and the execution data.

[0162] In one implementation, the processor 303 is further configured to:

[0163] Decompose the historical video stream into image frames arranged in time sequence; input the continuous image frames into the backbone network, extract the spatial dimension features, and obtain a spatial sequence composed of continuous spatial features; input the spatial sequence into the jump extraction branch to obtain the process prediction feature; input the spatial sequence into the comparison branch to obtain the comparison feature; determine the process corresponding to the image frame based on the spatiotemporal fusion features and the comparison features.

[0164] In one implementation, the processor 303 is further configured to:

[0165] The continuous image frames are convolutionally pooled to obtain pooled features; the pooled features are input into a feature extraction module to obtain extracted visual space features; the visual space features are globally pooled to obtain a spatial sequence composed of continuous space features.

[0166] In one implementation, the processor 303 is further configured to:

[0167] The spatial sequence is input into a single-layer LSTM network to obtain the motion information between consecutive frames; the motion information is mapped to obtain the first-level process prediction; the motion information is analyzed to obtain the second-level process prediction; corrections are made based on the first and second levels to determine the corresponding process.

[0168] In this application, Figure 4 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 4 Components shown.

[0169] The electronic device provided in this embodiment is based on the same inventive concept as the multi-robotic arm control method based on embodied intelligence provided in the embodiment of the present application, and has the same beneficial effects as the method adopted, run or implemented by the application stored in it.

[0170] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0171] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0173] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory (Flash RAM). The memory is an example of a computer-readable medium.

[0174] The present application also provides a computer-readable storage medium corresponding to the multi-robotic arm control method based on embodied intelligence provided in the aforementioned embodiment, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the interactive image analysis auxiliary method for 3D aerial imaging provided in any of the aforementioned embodiments.

[0175] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0176] The computer-readable storage medium provided in the above-mentioned embodiment of the present application and the interactive image analysis auxiliary method for 3D aerial imaging provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0177] It should be noted that in the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0178] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0179] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A multi-robot arm control method based on embodied intelligence, characterized in that: include: Get historical video streams of the same type of surgery; Parsing the historical video stream to obtain historical surgery information; Get the pending surgery information of the current object; Generate surgical planning and execution information for the current object based on historical surgical information, and control the multi-robot arm to execute the execution information.

2. The multi-robot control method based on embodied intelligence according to claim 1, characterized in that: The generating of the surgical planning and execution information of the current object based on the historical surgical information includes: Processing the historical surgical information based on inverse reinforcement learning to obtain a historical reward function; Construct multiple constraints; Based on multi-objective optimization, the execution strategy of multiple robotic arms is trained through reinforcement learning; After the training is completed, the execution information with the highest cumulative reward is selected and the multi-manipulator is controlled to execute the execution information.

3. The multi-robot control method based on embodied intelligence according to claim 2, characterized in that: The constraint conditions include obstacle constraints, and / or robot arm collision constraints, and / or safety boundary constraints.

4. The multi-robot control method based on embodied intelligence according to claim 1, characterized in that: The parsing of the historical video stream to obtain historical surgery information includes: Obtaining process information of the same type of surgery; Construct process conversion relationships at different levels and time granularities; Decomposing the historical video stream by spatial dimension and temporal dimension to identify each process in the historical video stream; Identify the end of the actuator in the process and determine the execution data of the surgical instrument; Based on the identified procedure and the execution data, historical surgical information is determined.

5. The multi-robot control method based on embodied intelligence according to claim 4, characterized in that: Decomposing the historical video stream by spatial dimension and temporal dimension to identify each process in the historical video stream includes: Decomposing historical video streams into time-sequentially arranged image frames; Inputting the continuous image frames into the backbone network, extracting spatial dimension features, and obtaining a spatial sequence composed of continuous spatial features; Input the spatial sequence into the jump extraction branch to obtain the process prediction feature; Input the spatial sequence into the contrast branch to obtain contrast features; According to the spatiotemporal fusion features and contrast features, the process corresponding to the image frame is determined.

6. The multi-robot control method based on embodied intelligence according to claim 5, characterized in that: The continuous image frames are input into the backbone network, and spatial dimension features are extracted to obtain a spatial sequence composed of continuous spatial features, including: Perform convolution pooling on the continuous image frames to obtain pooling features; Input the pooled features into the feature extraction module to obtain the extracted visual space features; The visual spatial features are globally pooled to obtain a spatial sequence consisting of continuous spatial features.

7. The multi-robot control method based on embodied intelligence according to claim 5, characterized in that: The step of inputting the spatial sequence into the jump extraction branch to obtain the process prediction feature comprises: Input the spatial sequence into a single-layer LSTM network to obtain the motion information between consecutive frames; Map the motion information to obtain the first-level process prediction; Analyze the motion information to obtain the second-level process prediction; Make corrections based on the first and second levels and determine the corresponding process.

8. A multi-robot arm control device based on embodied intelligence, characterized in that: include: A video acquisition module, which is used to acquire historical video streams of the same type of surgery; A video parsing module, which is used to parse the historical video stream to obtain historical surgery information; An information acquisition module, which is used to obtain the pending surgery information of the current object; The planning and execution module is used to generate the surgical planning and execution information of the current object based on the historical surgical information, and control the multi-manipulator to execute the execution information.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program to: Get historical video streams of the same type of surgery; Parsing the historical video stream to obtain historical surgery information; Get the pending surgery information of the current object; Generate surgical planning and execution information for the current object based on historical surgical information, and control the multi-robot arm to execute the execution information.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by the processor to implement the multi-robotic arm control method based on embodied intelligence as described in any one of claims 1-7.

Images