Target object segmentation method and device, electronic equipment and storage medium

Abstract

The application provides a target object segmentation method and device, electronic equipment and storage medium. The method comprises: acquiring a video frame containing a target object and a mask image; wherein the target object contains at least two parts that move relative to each other, and the mask image is rendered according to motion information of the target object collected synchronously with the video frame; performing feature extraction on the video frame and the mask image respectively to obtain first features and second features; processing the first features and the second features according to the connection relationship between the parts in the target object to obtain a first directed graph and a second directed graph; each of the first directed graph and the second directed graph includes nodes corresponding to the parts in the target object; and performing segmentation processing on the end part of the target object according to the first features, the second features, the first directed graph and the second directed graph to obtain an end part segmentation result of the target object. This is beneficial to improving the accuracy of the end part segmentation result.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to a target object segmentation method and device, an electronic device, and a computer readable storage medium. BACKGROUND

[0002] Semantic segmentation of different parts of a target object in an image is an important task in the field of computer vision. For example, in the medical field, it is necessary to segment the end part of a surgical instrument in an image to locate the contact between the end part of the surgical instrument and human tissue, thereby providing effective information for applications such as auxiliary perception of surgical state and early warning, visual control of automated surgery, etc. For another example, in the field of industrial automation, segmenting the end part of a robot or a mechanical arm from an image can help control and monitor its movement, achieving precise and safe operation and quality control. For yet another example, in the manufacturing field, segmenting different parts of a product from an image can help with assembly and repair operations, improving production efficiency and quality.

[0003] Although extensive research has been conducted on this task, the end part segmentation scheme of the target object in the related art is still not accurate enough for the segmentation of the end part of the target object. SUMMARY

[0004] Therefore, the present application provides a target object segmentation method and device, an electronic device, and a computer readable storage medium.

[0005] Specifically, the present application is implemented by the following technical solutions:

[0006] In a first aspect, the present application provides a target object segmentation method, comprising:

[0007] obtaining a video frame containing a target object and a mask image; wherein the target object contains at least two parts that move relative to each other, and the mask image is rendered according to motion information of the target object collected synchronously with the video frame;

[0008] performing feature extraction on the video frame and the mask image respectively to obtain first features and second features;

[0009] processing the first features and the second features respectively according to the connection relationship between the parts in the target object to obtain a first directed graph and a second directed graph; the first directed graph and the second directed graph each include nodes corresponding to the parts in the target object respectively;

[0010] performing segmentation processing on the end part of the target object according to the first features, the second features, the first directed graph, and the second directed graph to obtain an end part segmentation result of the target object.

[0011] In a second aspect, an embodiment of the present application provides a target object segmentation device, comprising:

[0012] An image acquisition module is configured to acquire a video frame containing a target object and a mask image, wherein the target object contains at least two parts moving relative to each other, and the mask image is rendered according to motion information of the target object acquired synchronously with the video frame;

[0013] A feature extraction module is configured to perform feature extraction on the video frame and the mask image respectively to obtain first features and second features;

[0014] A directed graph acquisition module is configured to process the first features and the second features respectively according to a connection relationship between the parts in the target object to obtain a first directed graph and a second directed graph, wherein the first directed graph and the second directed graph each include nodes corresponding to the parts in the target object respectively;

[0015] A segmentation processing module is configured to perform segmentation processing on an end part of the target object according to the first features, the second features, the first directed graph and the second directed graph to obtain an end part segmentation result of the target object.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method of the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium having a computer program stored thereon, wherein the program is executable by a processor to implement the method of the first aspect.

[0018] The technical scheme provided by the embodiments of the present application can have the following beneficial effects:

[0019] This method acquires video frames containing the target object and renders a mask image based on the motion information of the target object acquired synchronously with the video frames. Then, it processes the video frames and mask images to obtain a first feature, a second feature, a first directed graph, and a second directed graph. Each of the first and second directed graphs includes nodes corresponding to different parts of the target object. Finally, it segments the end parts of the target object based on the first feature, the second feature, the first directed graph, and the second directed graph to obtain the end part segmentation result. This method, by combining the motion information of the target object with the video frames containing the target object, achieves a more interpretable end part segmentation result. By constructing a directed graph to align the target object information in the video frames with the target object's motion information, it ensures robust performance of the end part segmentation result under different new application scenarios, improving the accuracy of the end part segmentation result. Attached Figure Description

[0020] Figure 1A This is a schematic diagram of various parts of a surgical instrument as illustrated in an exemplary embodiment of this application.

[0021] Figure 1B This is a schematic diagram of a robotic surgical system illustrated in an exemplary embodiment of this application.

[0022] Figure 1C This is a schematic diagram of a patient-side robot shown in an exemplary embodiment of this application.

[0023] Figure 2 This is a schematic diagram of a video frame and a mask image shown in an exemplary embodiment of this application.

[0024] Figure 3 This is a schematic diagram of the structure of an end-part segmentation model shown in an exemplary embodiment of this application.

[0025] Figure 4 This is a schematic diagram of another end-part segmentation model illustrated in an exemplary embodiment of this application.

[0026] Figure 5 This is a schematic diagram illustrating the input and output of a first graph neural network and a second graph neural network according to an exemplary embodiment of this application.

[0027] Figure 6 This is a schematic diagram of the structure of another end-part segmentation model shown in an exemplary embodiment of this application.

[0028] Figure 7 This is a schematic diagram illustrating contrastive learning using a contrastive loss function, as shown in an exemplary embodiment of this application.

[0029] Figure 8This is a flowchart illustrating a target object segmentation method according to an exemplary embodiment of this application.

[0030] Figure 9 This is a schematic diagram illustrating the segmentation result of the distal end of a surgical instrument according to an exemplary embodiment of this application.

[0031] Figure 10 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of this application.

[0032] Figure 11 This is a schematic diagram of the structure of a target object segmentation device shown in an exemplary embodiment of this application. Detailed Implementation

[0033] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0034] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0035] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0036] Semantic segmentation of different parts of a target object in an image is an important task in the field of computer vision. For example, in the medical field, segmenting the end parts of surgical instruments in an image is necessary to determine their contact with human tissue, thus providing valuable information for applications such as assisted perception of surgical status and early warning, and visual control of automated surgery. In industrial automation, segmenting the end parts of robots or robotic arms from images helps control and monitor their movement during industrial production, enabling precise and safe operation and quality control. In manufacturing, segmenting different parts of a product from images aids in assembly and maintenance operations, improving production efficiency and quality. Finally, in instrumental performance, segmenting different parts of a musical instrument (such as strings, keyboard, and drumhead) from images helps users learn and understand correct playing postures and techniques.

[0037] One approach to segmenting the distal ends of a target object in related technologies primarily involves designing a neural network structure to extract features from video frames or images containing the target object, thereby performing semantic segmentation. The neural network structure learns prior information about the target object from training samples, and then performs semantic segmentation based on this learned prior information. However, environmental interference varies in different application scenarios, and the learned prior information may not be applicable to new application scenarios, resulting in insufficient accuracy in the segmentation of the distal ends of the target object.

[0038] Taking the medical field as an example: the environmental characteristics of surgical instruments vary greatly in different surgical scenarios. For example, the tissue characteristics of different patients and different surgical sites are quite different. In addition, bloodstains, smoke, artifacts and other factors generated during the operation will also change the environmental characteristics of surgical instruments. As a result, the prior information originally learned by the neural network structure becomes invalid in the new surgical scenario. This leads to the lack of robustness and generalization of the originally trained neural network structure in the new surgical scenario, and the inaccurate segmentation of the end parts of the surgical instruments.

[0039] In response to the problems in the related technologies, the inventors considered that the target object has the same motion law in different application scenarios during the implementation of this application. The motion mode between at least two parts of the target object that move relative to each other is fixed. That is, the motion information of the target object will not be affected by changes in the application scenario. It can provide the necessary topological structure information and geometric prior.

[0040] Therefore, this application provides a target object segmentation method to obtain a video frame containing the target object and a mask image. The mask image is rendered based on the motion information of the target object synchronously acquired with the video frame. Then, feature extraction is performed on the video frame and the mask image to obtain a first feature and a second feature. Next, based on the connection relationships between different parts of the target object, the first feature and the second feature are processed to obtain a first directed graph and a second directed graph. Each of the first and second directed graphs includes nodes corresponding to different parts of the target object. Finally, the end parts of the target object are segmented based on the first feature, the second feature, the first directed graph, and the second directed graph to obtain the end part segmentation result. This method achieves a more interpretable end part segmentation result by combining the motion information of the target object with the video frame containing the target object. By constructing a directed graph to align the information of the target object in the video frame with the motion information of the target object, the end part segmentation result of the target object exhibits robust performance under different new application scenarios, improving the accuracy of the end part segmentation result.

[0041] For example, the target object segmentation method can be performed by an electronic device, including but not limited to a server, cloud server, smartphone / mobile phone, tablet computer, personal digital assistant (PDA), laptop computer, desktop computer, media content player, video game console / system, virtual reality system, augmented reality system, wearable device (e.g., watch, glasses, gloves, headwear (e.g., hat, helmet, virtual reality headset, augmented reality headset, head-mounted device (HMD), headband)) or any other type of device.

[0042] The target object segmentation method provided in this application can be applied to different fields. The medical field is used as an example below. It should be understood that the application scenarios illustrated below are merely illustrative and not intended to limit this disclosure.

[0043] In one embodiment, such as in the medical field, the target object includes at least one surgical instrument. For example, please refer to... Figure 1A As shown, the surgical instrument 001 may include a base 10, a wrist 20 (also referred to as a wrist joint), and an end effector 30. The wrist 20 connects the base 10 and the end effector 30, allowing the end effector 30 to move relative to the base 10. The end portion of the target object is the end effector 30. Figure 1AIn the example shown, the wrist 20 of the surgical instrument has two rotational joints, namely a pitch joint and a yaw joint, so the end effector 30 can perform pitch and yaw movements relative to the base 10. It is understood that the various parts of different types of surgical instruments may vary; for example, some surgical instruments may have a wrist 20 that includes only one rotational joint. The number of parts in the above example does not constitute a limitation on surgical instruments.

[0044] Surgical instruments include, but are not limited to: scalpels, used to cut and open tissues, with blades of different shapes and sizes suitable for different types of surgery; forceps and tweezers, used to grasp, hold, and move tissues, blood vessels, or other small structures, such as vascular forceps and tissue tweezers; dissectors, used to separate tissue layers, such as nerve dissectors and vascular dissectors; anastomosing devices, used to connect tissues or blood vessels together to promote healing, such as various types of suture needles and anastomosing devices; and suction devices, used to remove blood, body fluids, and other waste materials from the surgical area to keep the surgical area clean.

[0045] The end effector 30 includes, but is not limited to, hooks, shovels, needles, clamps, scissors, etc.

[0046] In one exemplary application scenario, robot-based minimally invasive surgery offers advantages such as accurate positioning, stable operation, minimal surgical trauma, and rapid postoperative recovery. Computer-aided surgical robot automation, as a crucial component of smart healthcare, can reduce the rate of surgical errors by doctors and decrease the burden of repetitive surgical tasks under fixed procedures. Among these, the segmentation of the distal ends of surgical instruments plays a vital role in robot-based minimally invasive surgery.

[0047] Surgical instruments can be held on the robotic arm of a surgical robot, and the robotic arm is equipped with sensors for acquiring motion information of the surgical instruments held on the robotic arm. These sensors include, but are not limited to, displacement sensors, orientation sensors, and / or position sensors. In one example, the robotic arm includes multiple sequentially connected connecting arms, with adjacent connecting arms connected by rotary joints. The surgical instruments are mounted on the end-effector connecting arm. Sensors may include encoders located at each rotary joint, displacement sensors located on the linear drive module of the end-effector connecting arm, and encoders located on the tool drive module of the end-effector connecting arm. The surgical robot is also equipped with an image acquisition device, such as an endoscope held on the robotic arm, for acquiring video frames containing the surgical instruments during surgery. Based on the video frames containing the surgical instruments and the motion information of the surgical instruments acquired synchronously with the video frames, the target object segmentation method provided in this application can be used to obtain the segmentation result of the end portion of the surgical instruments, thereby providing necessary visual servo feedback, surgical status analysis, and surgical hazard warnings for automated surgery.

[0048] For example, please refer to Figure 1B This is a schematic diagram of a robotic surgical system 100. During surgery, the patient is positioned in front of a patient-side robot (PSR) 101, which includes one or more robotic arms 101a. The end of each robotic arm 101a is used to hold one or more surgical instruments 001. The surgeon can control the robotic arms 101a via a surgeon console (SGC) 102, thereby controlling the surgical instruments 001 to perform surgical procedures on the patient. The robotic arms 101a can also hold image acquisition devices (e.g., endoscopes, not shown). The surgeon can control the robotic arm 101a holding the endoscope via the surgeon console 102, thereby moving and holding the endoscope near the patient's lesion area to acquire surgical images including the patient's lesion and surrounding tissues, and the surgical instruments 001. During the surgery, the surgical instruments 001 and / or endoscope on the robotic arm 101a are inserted into the patient's body through a pre-set incision and can rotate around the center point of that incision (generally referred to as the Remote Center of Motionpoint, RCM). The images acquired by the endoscope are transmitted to the Vision Cart (VCT) 103 for image processing and recording, and the processed images are displayed on the respective display devices of the Vision Cart 103 and the doctor's console 102 for observation by the doctor and other surgical staff.

[0049] For example, please refer to Figure 1C A schematic diagram of a patient-side robot 101 is shown. The patient-side robot 101 includes a chassis 101b, a push handle 101c, and at least one robotic arm 101a (only one robotic arm 101a is shown for illustration purposes). Each robotic arm 101a includes an adjustment arm 101a-1 and an operating arm 101a-2. The robotic arm 101a-2 includes one or more sensors, such as displacement sensors, orientation sensors, and / or position sensors. The kinematic data of the robotic arm 101a and the surgical instrument 001 held by the robotic arm 101a can be obtained through the detection values ​​of these sensors; for example, the pose information of various parts of the surgical instrument 001.

[0050] In some embodiments, the target object segmentation method provided in this application is implemented using an end-part segmentation model. The training process of the end-part segmentation model can be performed by an electronic device. The training process of the end-part segmentation model is described below by way of example:

[0051] In some embodiments, the training process of the end segmentation model may be as follows: first, a model is represented by modeling; then, an evaluation function is constructed to evaluate the model; finally, the evaluation function is optimized based on sample data and optimization methods to adjust the model to its optimal state.

[0052] 1. Modeling: During training, the first step is to select a model architecture suitable for the task. The model can be a neural network, decision tree, support vector machine, etc. The goal of modeling is to select a model that can effectively fit the data and has appropriate complexity.

[0053] 2. Evaluation Function: During training, an evaluation function needs to be defined to measure the difference or error between the model's predictions and the true labels. This evaluation function is usually called the loss function, and it measures the model's performance.

[0054] 3. Optimization Method: During training, an optimization method needs to be selected to minimize the loss function. Common optimization methods include Stochastic Gradient Descent (SGD), Adam, and RMSprop. These methods improve the model's predictive ability by adjusting the model parameters to gradually reduce the loss function.

[0055] 4. Training Data: Model training requires a large amount of training data. This data is typically divided into a training set, a validation set, and a test set. The training set is used for parameter updates, the validation set is used for tuning model hyperparameters, and the test set is used to evaluate the model's generalization ability on unseen data.

[0056] 5. Training Iterations: During model training, model parameters are gradually adjusted through iterative optimization. Each training iteration includes steps such as forward propagation, calculating the loss function value, backward propagation, and parameter updates. The number of training iterations is usually set to a fixed number or dynamically adjusted based on the model's performance on the validation set.

[0057] Through the above steps, the model can be trained using sample data and, after multiple iterations of optimization, ultimately yield an optimal model capable of accurate prediction and generalization. Note that during model training, techniques such as overfitting control, learning rate adjustment, and regularization are also necessary to improve the model's training effectiveness and generalization ability.

[0058] In some embodiments, the electronic device can perform supervised training on the end-part segmentation model based on input samples and their labels; the input samples include video frames containing the target object and a mask image, the mask image being rendered based on motion information synchronously acquired with the video frame; the labels include labeled images of the end-parts of the target object in the video frame and labeled images of other parts.

[0059] (1) The preparation of training samples is an important aspect of the training process of the end segmentation model.

[0060] In some embodiments, an image acquisition device can be used to acquire video frames containing the target object in different application scenarios. At the same time, a motion sensor is used to synchronously acquire the motion information of the target object during the acquisition of the video frames. Based on the acquired motion information of the target object, a mask map of the target object corresponding to the video frame is rendered.

[0061] For example, in the medical field, please refer to... Figure 2 The video frame containing the target object can be a video frame of surgical instrument operation collected by endoscope in different surgical scenarios. The motion information of the target object can be calculated from the motion information of the surgical instrument as a whole and its parts, which are measured synchronously by sensors during the acquisition of video frames. This information is used to render a surgical instrument mask map corresponding to the video frame.

[0062] The motion information of the target object intuitively reflects the topological logic and geometric morphology between its various parts, and this information can be adequately represented by a rendering mask. The process of obtaining the mask image is illustrated here: Please refer to [link to documentation]. Figure 2 The electronic device can acquire motion information of the target object synchronously acquired with the video frame, intrinsic and extrinsic parameters of the image acquisition device when acquiring the video frame, and three-dimensional graphics of the target object; then, it performs rendering processing based on the intrinsic and extrinsic parameters, the motion information, and the three-dimensional graphics to obtain the mask image.

[0063] The motion information of the target object includes at least one of the connection relationships between different parts of the target object and the pose information of each part. During the process of obtaining the mask image, the motion information can be used to transform different parts of the 3D graphic to obtain a transformed 3D graphic. Then, using the intrinsic and extrinsic parameters of the image acquisition device when acquiring the video frame as a reference, the transformed 3D graphic is mapped from 3D space to 2D space, and the mapping result is rendered to obtain the mask image of the target object. Please refer to [link to relevant documentation]. Figure 2 In this embodiment, a mask image obtained based on motion information is used. The motion state of the target object in the mask image is consistent with the motion state of the target object in the synchronously acquired video frame. Moreover, the target object in the mask image is not affected by the application scenario. The end-part segmentation model is trained by combining the mask image obtained based on motion information. This ensures that the trained end-part segmentation model is not affected by the application scenario, thus guaranteeing that the end-part segmentation results of the target object have robust performance under different new application scenarios and improving the accuracy of the end-part segmentation results of the target object.

[0064] In one possible implementation, the 3D graphics include a 3D mesh. To improve processing speed and meet real-time requirements, after transforming various parts of the 3D graphics using the motion information to obtain a transformed 3D graphics, the transformed 3D graphics can be further processed by mesh extraction to obtain an extracted 3D graphics. The extracted 3D graphics have fewer meshes than the transformed 3D graphics. Then, using the intrinsic and extrinsic parameters of the image acquisition device when acquiring the video frames as a reference, the extracted 3D graphics are mapped from 3D space to 2D space, and the mapping result is rendered to obtain an initial mask. Finally, based on the size relationship between the transformed 3D graphics and the extracted 3D graphics, the initial mask is enlarged to obtain the final mask. This embodiment employs a mesh extraction and scale contraction strategy to improve the rendering speed of the mask, thereby meeting the real-time requirements in practical scenarios.

[0065] In the network extraction process, the number of extracted meshes is positively correlated with the rendering speed required in the actual scene; that is, the faster the rendering speed, the more meshes need to be extracted.

[0066] For example, in order for the end-part segmentation model to accurately segment different parts of a target object, the different parts of the target object can be presented as at least one of different colors and different textures in the 3D graphics and the mask image. This allows the end-part segmentation model to more quickly and accurately identify different parts of the target object based on different colors and / or different textures during the training process, thereby improving the recognition accuracy.

[0067] (2) The specific structural design of the end-effector segmentation model is another important aspect of the training process. To enable the end-effector segmentation model to learn the relationship between visual and motion information, embodiments of this application provide, for example... Figures 3 to 6 The end segmentation model shown.

[0068] Please see Figure 3 The end segmentation model includes a first feature extraction network, a second feature extraction network, a first graph neural network connected to the first feature extraction network, a second graph neural network connected to the second feature extraction network, and a prediction network.

[0069] The first feature extraction network is used to extract features from the input video frames containing the target object to obtain the first feature.

[0070] The second feature extraction network is used to extract features from the mask image of the input target object to obtain the second feature.

[0071] The first graph neural network is used to process the first feature to obtain a first directed graph. The first directed graph includes nodes corresponding to each part of the target object; each node in the first directed graph contains features of the part of the target object corresponding to that node, and the connection relationships between the nodes in the first directed graph are determined based on the connection relationships between the parts of the target object.

[0072] The second graph neural network is used to process the second feature to obtain a second directed graph. The second directed graph includes nodes corresponding to each part of the target object; each node in the second directed graph contains features of the part of the target object corresponding to that node, and the connection relationships between the nodes in the second directed graph are determined based on the connection relationships between the parts of the target object.

[0073] In this embodiment, a first graph neural network and a second graph neural network are used to model the relationships between different parts of the target object. A directed graph is also established to transmit features along geometric connections from the base to the distal end of the target object. Taking surgical instruments in the medical field as an example, the relationship between three parts of the surgical instrument can be modeled: the distal end, the wrist, and the base. By establishing a directed graph, features are transmitted along geometric connections from the base of the surgical instrument along the wrist and then to the distal end. Such a model can effectively and explicitly model the relationships between different parts of a surgical instrument.

[0074] The prediction network is used to segment the end portion of the target object based on the first feature, the second feature, the first directed graph, and the second directed graph to obtain the segmentation result of the end portion of the target object.

[0075] Please see Figure 4 Here, the first feature extraction network and the second feature extraction network are illustrated by example:

[0076] The first feature extraction network includes a first encoder and a first converter network; the first encoder is used to extract a first local feature from the video frame, and the first converter network is used to extract a first non-local feature from the first local feature output by the first encoder.

[0077] For example, if the first non-local feature relates to global information of a video frame and includes features of various parts of a target object, and the target object includes at least two parts that move relative to each other, then the first non-local feature includes features of at least two channels, and the features of each channel correspond one-to-one with the various parts of the target object.

[0078] The second feature extraction network includes a second encoder and a second converter network. The second encoder is used to extract second local features from the mask image, and the second converter network is used to extract second non-local features from the second local features output by the second encoder. Similarly, the second non-local features involve global information of the mask image, containing features of various parts of the target object. The second non-local features include features from at least two channels, and the features of each channel correspond one-to-one with the various parts of the target object.

[0079] Local features refer to the feature information of a small region or local area in an image. Non-local features refer to the feature representations that are analyzed and extracted from the entire image. They usually cover the global statistical information of the entire image and can reflect the general features and structure of the entire image.

[0080] It is understood that the embodiments of this application do not impose any restrictions on the specific structures of the first encoder, the first converter network, the second encoder, and the second converter network, and can be specifically set according to the actual application scenario. For example, both the first encoder and the second encoder are convolutional neural network models (CNNs). Due to the limitation of the convolution kernel, the receptive field of a CNN model is limited, and it is mainly used to extract features within a limited adjacent spatial range to obtain local features. These local features mainly describe the texture, color, or shape within the receptive field region. Both the first converter network and the second converter network are attention-based neural network models (such as the Transformer model), which can extract non-local features, such as color distribution features, texture statistical features, shape descriptions, etc.

[0081] Here, we provide an exemplary illustration of the first and second graph neural networks:

[0082] To achieve accurate segmentation of the end portions of a target object, a graph neural network is constructed to model the relationships between the various parts of the target object. The first graph neural network processes the first feature to obtain a first directed graph. The second graph neural network processes the second feature to obtain a second directed graph. Through the first and second graph neural networks, the relationships between the various parts of the target object are modeled, and directed graphs are established to propagate features along geometric connections from the base to the end portions of the target object.

[0083] In one possible implementation, both the first graph neural network and the second graph neural network include at least two graph convolutional network layers connected sequentially. It is understood that this application embodiment does not impose any limitation on the specific number of graph convolutional network layers, and can be specifically set according to the actual application scenario.

[0084] The first graph convolutional network layer in the first graph neural network is used to perform convolution operations on the features related to each part of the target object contained in the first feature to obtain the convolution operation result; and, the non-first graph convolutional network layers in the first graph neural network layer are used to perform convolution operations again on the convolution operation result output by the previous graph convolutional network layer.

[0085] The first graph convolutional network layer in the second graph neural network is used to perform convolution operations on the features related to each part of the target object contained in the second feature to obtain the convolution operation result; and the non-first graph convolutional network layers in the second graph neural network layer are used to perform convolution operations again on the convolution operation result output by the previous graph convolutional network layer.

[0086] In one possible implementation, as mentioned above, the first non-local feature includes features from at least two channels, with each channel's feature corresponding one-to-one with a different part of the target object. Similarly, the second non-local feature includes features from at least two channels, with each channel's feature corresponding one-to-one with a different part of the target object. Figure 5 Taking the surgical instrument shown as an example, the first non-local feature includes the features of three channels, which correspond one-to-one with the base, wrist and end effector of the surgical instrument; similarly, the second non-local feature includes the features of three channels, which correspond one-to-one with the base, wrist and end effector of the surgical instrument.

[0087] The first graph neural network is used to process the first non-local features to obtain a first directed graph. The first directed graph includes v1 node, v2 node and v3 node. v1 node represents the end effector of the surgical instrument, v2 node represents the wrist of the surgical instrument and v3 node represents the base of the surgical instrument.

[0088] The second graph neural network is used to process the second nonlocal feature to obtain a second directed graph, which includes nodes k1, k2, and k3. Node k1 represents the end effector of the surgical instrument, node k2 represents the wrist of the surgical instrument, and node k3 represents the base of the surgical instrument.

[0089] As mentioned above, both the first and second graph neural networks contain at least two sequentially connected graph convolutional network layers. That is, the data input to the first graph convolutional network layer of the first graph neural network is the first non-local feature; the data input to the first graph convolutional network layer of the second graph neural network is the second non-local feature.

[0090] Please see Figure 6 Here is an example of a prediction network:

[0091] The prediction network includes a splicing layer and a decoder. See also... Figure 6To avoid the gradual loss of shallow information as the number of layers increases in deep networks, the concatenation layer is used to concatenate the first non-local feature, the second non-local feature, the first directed graph, and the second directed graph to obtain concatenated features. Furthermore, considering that the first local feature output by the first encoder is similar to the segmentation target of the end-part segmentation model, to avoid the gradual loss of shallow information, a skip connection is established between the first encoder and the decoder. The decoder is used to perform prediction processing on the concatenated features and the first local feature to predict the end-part segmentation result of the target object. That is, the first feature may include the first local feature and the first non-local feature; the second feature may include the second non-local feature.

[0092] (3) Another important aspect of the training process is designing a suitable loss function based on business needs. The loss function, also known as the cost function, is used in supervised model training scenarios where the sample data contains labeled true values. It measures the error between the model's predicted values ​​and the true values. The loss function is crucial to the model's accuracy, and designing a suitable loss function based on existing sample data and the model's requirements is quite challenging.

[0093] Through the first and second graph neural networks, the end-part segmentation model can learn the distribution and connectivity of different parts of the target object during training using graph network contrastive learning. Therefore, please refer to... Figure 7 , Figure 7 Taking the first and second directed graphs of surgical instruments as examples, the loss function used by the end-part segmentation model during training includes at least: a contrastive loss function for optimizing the first and second graph neural networks; the optimization objective of the contrastive loss function includes: in the first and second directed graphs obtained based on input samples, the features of two nodes representing the same part of the target object are the same or similar, and the features of two nodes representing different parts of the target object are different. Specifically, "the features of two nodes representing the same part of the target object are the same or similar" optimizes the features of a single part of the target object, enabling the end-part segmentation model to acquire motion information as prior. "The features of two nodes representing different parts of the target object are different" optimizes the features between parts of the target object, resulting in more explicit segmentation results between different parts.

[0094] The end-part segmentation model can also learn prior segmentation information for each part of the target object through supervised learning during training. As mentioned above, the electronic device can perform supervised training on the end-part segmentation model based on input samples and their labels; the input samples include video frames containing the target object and a mask image, the mask image being rendered based on motion information synchronously acquired with the video frames; the labels include labeled images of the end parts of the target object and labeled images of other parts in the video frames.

[0095] Therefore, the loss function used in the training process of the end segmentation model may also include a first loss function for optimizing the end segmentation model and a second loss function for optimizing the first converter network.

[0096] The optimization objective of the first loss function includes minimizing the difference between the predicted segmentation result output by the end-part segmentation model during training and the end-part labeled image.

[0097] The optimization objective of the second loss function includes minimizing the difference between the predicted segmentation results of each part of the target object and the labeled images of each part of the target object; wherein the predicted segmentation results of each part of the target object are obtained based on the first non-local features output by the first converter network. The first non-local features contain features of at least two channels, and the features of each channel correspond one-to-one with each part of the target object. Therefore, the features of each channel of the first non-local features can be decoded to obtain the predicted segmentation results of each part of the target object. The purpose of designing the second loss function in this embodiment is to ensure that the features of each channel output by the first converter network can accurately correspond to each part of the target object. It is understood that this embodiment does not impose any restrictions on the specific form of the loss function, and it can be specifically set according to the actual application scenario. For example, the first loss function and the second loss function can be at least one of the Dice loss function and the cross-entropy loss function. The Dice loss function is a loss function used for image segmentation tasks, especially suitable for handling unbalanced and unclear boundary segmentation problems. The Dice coefficient can measure the similarity between the predicted result and the real label. The cross-entropy loss function is widely used in tasks such as image classification and object detection. The cross-entropy can measure the difference between the predicted result and the real label. However, it is not limited to this; it can also be the mean squared error loss function, the logarithmic loss function, and so on.

[0098] The training process is an iterative optimization process. A well-trained segmentation model for the terminal limbs can be obtained when the iteration termination condition is met. This termination condition could be that the loss function of the segmentation model reaches the optimization objective, or the number of iterations reaches a preset number, etc.

[0099] The following is an example illustrating the application process of the end-part segmentation model:

[0100] Please see Figure 8 This application provides a flowchart illustrating a target object segmentation method. The method includes:

[0101] In S101, a video frame containing a target object and a mask image are acquired; wherein the target object contains at least two parts that move relative to each other, and the mask image is rendered based on the motion information of the target object acquired synchronously with the video frame.

[0102] In S102, feature extraction is performed on the video frame and the mask image respectively to obtain the first feature and the second feature.

[0103] In S103, the first feature and the second feature are processed according to the connection relationship between the parts of the target object to obtain a first directed graph and a second directed graph; each of the first directed graph and the second directed graph includes nodes corresponding to the parts of the target object.

[0104] In S104, the end portion of the target object is segmented based on the first feature, the second feature, the first directed graph, and the second directed graph to obtain the end portion segmentation result of the target object.

[0105] In this embodiment, considering that the motion information of the target object is not affected by changes in the application scenario, it can provide the necessary topological structure information and geometric prior. Based on video frames containing the target object, its motion information is combined to obtain a more interpretable end-part segmentation result. A directed graph aligns the target object information in the video frame with its motion information, ensuring robust performance of the end-part segmentation result under different new application scenarios and improving the accuracy of the end-part segmentation result.

[0106] In some embodiments, for S101, the process of obtaining the mask image is illustrated below: The electronic device can acquire the motion information of the target object synchronously acquired with the video frame, the intrinsic and extrinsic parameters of the image acquisition device when acquiring the video frame, and the three-dimensional graphic of the target object; the mask image is obtained by rendering based on the intrinsic and extrinsic parameters, the motion information, and the three-dimensional graphic. The motion information of the target object intuitively reflects the topological logic information and geometric morphological information between various parts of the target object, and the rendering mask is sufficient to represent this information.

[0107] The motion information of the target object includes at least one of the connection relationships between different parts of the target object and the pose information of each part. During the process of obtaining the mask, the motion information can be used to transform different parts of the 3D graphic to obtain a transformed 3D graphic. Then, using the intrinsic and extrinsic parameters of the image acquisition device when acquiring the video frame as a reference, the transformed 3D graphic is mapped from 3D space to 2D space, and the mapping result is rendered to obtain the mask of the target object. In this embodiment, the mask obtained based on motion information ensures that the motion state of the target object in the mask is consistent with the motion state of the target object in the synchronously acquired video frame, and the target object in the mask is unaffected by the application scenario. This guarantees robust performance of the segmentation results of the target object's end parts under different new application scenarios, improving the accuracy of the segmentation results of the target object's end parts.

[0108] In one possible implementation, to improve processing speed and meet real-time requirements, after transforming various parts of the 3D graphic using the motion information to obtain a transformed 3D graphic, the transformed 3D graphic can be further processed by mesh extraction to obtain an extracted 3D graphic, where the number of meshes in the extracted 3D graphic is less than the number of meshes in the transformed 3D graphic. Then, using the intrinsic and extrinsic parameters of the image acquisition device when acquiring the video frames as a reference, the extracted 3D graphic is mapped from 3D space to 2D space, and the mapping result is rendered to obtain an initial mask image. Finally, based on the size relationship between the transformed 3D graphic and the extracted 3D graphic, the initial mask image is enlarged to obtain the final mask image. This embodiment employs a mesh extraction and scale contraction strategy to improve the rendering speed of the mask image, meeting the real-time requirements of practical scenarios.

[0109] For example, in order for the end-part segmentation model to accurately segment different parts of the target object, the different parts of the target object can be presented as at least one of different colors and different textures in the 3D graphics and the mask image, so that the end-part segmentation model can more quickly and accurately identify different parts of the target object based on different colors and / or different textures, thereby improving the recognition accuracy.

[0110] Please see Figures 4 to 6 In some embodiments, for S102, the electronic device can input the video frame and the mask image into a pre-trained end-part segmentation model; extract features from the video frame through a first feature extraction network in the end-part segmentation model to obtain a first feature; and extract features from the mask image through a second feature extraction network in the end-part segmentation model to obtain a second feature.

[0111] For example, the first feature extraction network includes a first encoder and a first converter network; the first encoder is used to extract a first local feature from the video frame, and the first converter network is used to extract a first non-local feature from the first local feature output by the first encoder.

[0112] The second feature extraction network includes a second encoder and a second converter network; the second encoder is used to extract a second local feature from the mask image, and the second converter network is used to extract a second non-local feature from the second local feature output by the second encoder.

[0113] In some embodiments, for S103, the first feature can be processed by a first graph neural network in the end-part segmentation model to obtain a first directed graph; each node in the first directed graph contains features of the part of the target object corresponding to that node, and the connection relationship between each node in the first directed graph is determined according to the connection relationship between each part of the target object; and the second feature can be processed by a second graph neural network in the end-part segmentation model to obtain a second directed graph; each node in the second directed graph contains features of the part of the target object corresponding to that node, and the connection relationship between each node in the second directed graph is determined according to the connection relationship between each part of the target object. In this embodiment, the part relationships of each part of the target object are modeled by the first graph neural network and the second graph neural network, and a directed graph is established to propagate features from the base of the target object to the end part along geometric connections.

[0114] Both the first graph neural network and the second graph neural network contain at least two graph convolutional network layers connected in sequence.

[0115] The first graph convolutional network layer in the first graph neural network is used to perform convolution operations on the features related to each part of the target object contained in the first feature to obtain the convolution operation result; and, the non-first graph convolutional network layers in the first graph neural network layer are used to perform convolution operations again on the convolution operation result output by the previous graph convolutional network layer.

[0116] The first graph convolutional network layer in the second graph neural network is used to perform convolution operations on the features related to each part of the target object contained in the second feature to obtain the convolution operation result; and the non-first graph convolutional network layers in the second graph neural network layer are used to perform convolution operations again on the convolution operation result output by the previous graph convolutional network layer.

[0117] For example, the first directed graph is obtained by processing the first non-local feature through a first graph neural network in the end-part segmentation model; the second directed graph is obtained by processing the second non-local feature through a second graph neural network in the end-part segmentation model. That is, the data input to the first graph convolutional network layer in the first graph neural network is the first non-local feature; the data input to the first graph convolutional network layer in the second graph neural network is the second non-local feature.

[0118] In some embodiments, for S104, the end-part segmentation result of the target object can be obtained through the prediction network in the end-part segmentation model. Specifically, the prediction network includes a stitching layer and an encoder. The stitching layer can stitch together the first non-local feature, the second non-local feature, the first directed graph, and the second directed graph to obtain stitched features. Then, the decoder performs prediction processing on the stitched features and the first local feature to predict the end-part segmentation result of the target object. For example, please refer to [link to example]. Figure 9 , Figure 9 The image shows a video frame containing surgical instruments. After processing the video frame using the target object segmentation method described above, the image shows the segmentation result of the end portion of the surgical instruments.

[0119] This embodiment achieves a more interpretable end-part segmentation result by combining the motion information of the target object with the video frame containing the target object. By constructing a directed graph, the information of the target object in the video frame is aligned with the motion information of the target object, ensuring that the end-part segmentation result of the target object has robust performance in different new application scenarios and improving the accuracy of the end-part segmentation result of the target object.

[0120] The various technical features in the above embodiments can be combined arbitrarily, as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of various technical features in the above embodiments is also within the scope of this specification.

[0121] Corresponding to the aforementioned embodiments of the target object segmentation method, this application also provides embodiments of the target object segmentation apparatus.

[0122] The embodiments of the target object segmentation device of this application can be applied to electronic devices. The device embodiments can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of the electronic device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 10The diagram shown is a hardware structure diagram of an electronic device containing the target object segmentation device of this application, except... Figure 10 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in which the device is located in the embodiment may also include other hardware, depending on the actual function of the target object, which will not be described in detail here.

[0123] Please refer to Figure 11 This application provides a target object segmentation device, including:

[0124] Image acquisition module 201 is used to acquire a video frame containing a target object and a mask image; wherein, the target object contains at least two parts that move relative to each other, and the mask image is rendered based on the motion information of the target object acquired synchronously with the video frame;

[0125] Feature extraction module 202 is used to extract features from the video frame and the mask image respectively to obtain a first feature and a second feature;

[0126] The directed graph acquisition module 203 is used to process the first feature and the second feature respectively according to the connection relationship between each part of the target object to obtain a first directed graph and a second directed graph; the first directed graph and the second directed graph each include nodes corresponding to each part of the target object respectively;

[0127] The segmentation processing module 204 is used to segment the end portion of the target object according to the first feature, the second feature, the first directed graph and the second directed graph to obtain the segmentation result of the end portion of the target object.

[0128] In some embodiments, the image acquisition module 201 includes:

[0129] The data acquisition unit is used to acquire motion information of the target object that is synchronously acquired with the video frame, internal and external parameters of the image acquisition device when acquiring the video frame, and three-dimensional graphics of the target object;

[0130] The rendering unit is used to perform rendering processing based on the intrinsic and extrinsic parameters, the motion information, and the three-dimensional graphics to obtain the mask image.

[0131] In some embodiments, the motion information of the target object includes the connection relationships between different parts of the target object and / or the pose information of each part.

[0132] The rendering unit is specifically used to transform each part of the three-dimensional graphic using the motion information to obtain the transformed three-dimensional graphic; with reference to the internal and external parameters of the image acquisition device when acquiring the video frame, the transformed three-dimensional graphic is mapped from three-dimensional space to two-dimensional space, and the mapping result is rendered to obtain the mask image.

[0133] In some embodiments, the three-dimensional graphic includes a three-dimensional mesh; the rendering unit is specifically configured to: transform each part of the three-dimensional graphic using the motion information to obtain a transformed three-dimensional graphic; perform mesh extraction processing on the transformed three-dimensional graphic to obtain an extracted three-dimensional graphic; wherein the number of meshes in the extracted three-dimensional graphic is less than the number of meshes in the transformed three-dimensional graphic; using the intrinsic and extrinsic parameters of the image acquisition device when acquiring the video frame as a reference, map the extracted three-dimensional graphic from three-dimensional space to two-dimensional space, and perform rendering processing on the mapping result to obtain an initial mask image; and enlarge the initial mask image according to the size relationship between the transformed three-dimensional graphic and the extracted three-dimensional graphic to obtain the mask image.

[0134] In some embodiments, different parts of the target object are displayed with different colors and / or different textures in the three-dimensional graphics and the mask image.

[0135] In some embodiments, the feature extraction module 202 is specifically used to: input the video frame and the mask image into a pre-trained end-part segmentation model; extract features from the video frame through a first feature extraction network in the end-part segmentation model to obtain a first feature; and extract features from the mask image through a second feature extraction network in the end-part segmentation model to obtain a second feature.

[0136] In some embodiments, the directed graph acquisition module 203 is specifically used to: process the first feature through a first graph neural network in the end-part segmentation model to obtain a first directed graph; each node in the first directed graph contains features of the part of the target object corresponding to that node, and the connection relationship between each node in the first directed graph is determined according to the connection relationship between each part of the target object; and process the second feature through a second graph neural network in the end-part segmentation model to obtain a second directed graph; each node in the second directed graph contains features of the part of the target object corresponding to that node, and the connection relationship between each node in the second directed graph is determined according to the connection relationship between each part of the target object.

[0137] In some embodiments, the end-part segmentation model learns the distribution and connection relationships of each part in the target object through graph network contrastive learning during training; the loss function used by the end-part segmentation model during training includes at least: a contrastive loss function for optimizing the first graph neural network and the second graph neural network; the optimization objective of the contrastive loss function includes: in the first directed graph and the second directed graph obtained based on the input samples, the features of two nodes representing the same part in the target object are the same or similar, and the features of two nodes representing different parts in the target object are different.

[0138] In some embodiments, both the first graph neural network and the second graph neural network include at least two graph convolutional network layers connected in sequence; the first graph convolutional network layer in the first graph neural network is used to perform convolution operations on the features related to each part of the target object included in the first feature to obtain a convolution operation result; and a non-first graph convolutional network layer in the first graph neural network layer is used to perform a convolution operation again on the convolution operation result output by the previous graph convolutional network layer; the first graph convolutional network layer in the second graph neural network is used to perform convolution operations on the features related to each part of the target object included in the second feature to obtain a convolution operation result; and a non-first graph convolutional network layer in the second graph neural network layer is used to perform a convolution operation again on the convolution operation result output by the previous graph convolutional network layer.

[0139] In some embodiments, the first feature extraction network includes a first encoder and a first converter network; the first encoder is used to extract a first local feature from the video frame, and the first converter network is used to extract a first non-local feature from the first local feature output by the first encoder; the second feature extraction network includes a second encoder and a second converter network; the second encoder is used to extract a second local feature from the mask image, and the second converter network is used to extract a second non-local feature from the second local feature output by the second encoder.

[0140] In some embodiments, the first directed graph is obtained by processing the first nonlocal feature through a first graph neural network in the end-part segmentation model; the second directed graph is obtained by processing the second nonlocal feature through a second graph neural network in the end-part segmentation model.

[0141] In some embodiments, both the first encoder and the second encoder are convolutional neural network models; both the first converter network and the second converter network are attention-based neural network models.

[0142] In some embodiments, the segmentation processing module 204 is specifically used to: perform splicing processing on the first non-local feature, the second non-local feature, the first directed graph, and the second directed graph through the splicing layer in the end-part segmentation model to obtain spliced ​​features; and perform prediction processing on the spliced ​​features and the first local feature through the decoder in the end-part segmentation model to predict the end-part segmentation result of the target object.

[0143] In some embodiments, the end-part segmentation model is obtained through supervised training based on input samples and their labels; the input samples include video frames containing the target object and a mask image, the mask image being rendered based on motion information acquired synchronously with the video frame; the labels include labeled images of the end-parts of the target object in the video frame and labeled images of other parts.

[0144] The loss function used in the training process of the end-part segmentation model further includes: a first loss function and a second loss function; the optimization objective of the first loss function includes: minimizing the difference between the predicted segmentation result output by the end-part segmentation model during training and the labeled image of the end part; the optimization objective of the second loss function includes: minimizing the difference between the predicted segmentation result of each part of the target object and the labeled image of each part of the target object; wherein, the predicted segmentation result of each part of the target object is obtained based on the first non-local feature output by the first converter network.

[0145] In some embodiments, the target object includes at least one surgical instrument held on the robotic arm of a surgical robot, and the robotic arm is equipped with a motion sensor for acquiring motion information of the surgical instrument held on the robotic arm.

[0146] In some embodiments, the surgical instrument includes a base, a wrist, and an end effector, the wrist connecting the base and the end effector respectively, such that the end effector is movable relative to the base; the end part of the target object is the end effector.

[0147] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0148] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0149] In some embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any of the above embodiments.

[0150] In some embodiments, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the method as described in any of the preceding embodiments.

[0151] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0152] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0153] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0154] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0155] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0156] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0157] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0158] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A target object segmentation method, characterized by, The method comprises: obtaining a video frame containing a target object and a mask image; wherein the target object contains at least two parts moving relative to each other, and the mask image is rendered according to motion information of the target object collected synchronously with the video frame; performing feature extraction on the video frame and the mask image respectively to obtain first features and second features; processing the first features and the second features respectively according to the connection relationship between the parts in the target object to obtain a first directed graph and a second directed graph; the first directed graph and the second directed graph each include nodes corresponding to the parts in the target object respectively; performing segmentation processing on the end part of the target object according to the first features, the second features, the first directed graph and the second directed graph to obtain an end part segmentation result of the target object.

2. The method of claim 1, wherein, The method for obtaining a mask image comprises: obtaining motion information of the target object collected synchronously with the video frame, internal and external parameters of an image collection device when collecting the video frame, and a three-dimensional graph of the target object; performing rendering processing according to the internal and external parameters, the motion information and the three-dimensional graph to obtain the mask image.

3. The method of claim 2, wherein, The motion information of the target object includes the connection relationship between the parts in the target object and / or the pose information of the parts; The rendering processing according to the internal and external parameters, the motion information and the three-dimensional graph to obtain the mask image comprises: transforming the parts in the three-dimensional graph using the motion information to obtain a transformed three-dimensional graph; mapping the transformed three-dimensional graph from a three-dimensional space to a two-dimensional space with reference to the internal and external parameters of the image collection device when collecting the video frame, and performing rendering processing on the mapped result to obtain the mask image.

4. The method of claim 3, wherein, The three-dimensional graph comprises a three-dimensional mesh; After the transformation of the parts in the three-dimensional graph using the motion information to obtain a transformed three-dimensional graph, the method further comprises: performing mesh extraction processing on the transformed three-dimensional graph to obtain an extracted three-dimensional graph; wherein the number of meshes of the extracted three-dimensional graph is less than the number of meshes of the transformed three-dimensional graph; mapping the extracted three-dimensional graph from a three-dimensional space to a two-dimensional space with reference to the internal and external parameters of the image collection device when collecting the video frame, and performing rendering processing on the mapped result to obtain the mask image, comprising: mapping the extracted three-dimensional graph from a three-dimensional space to a two-dimensional space with reference to the internal and external parameters of the image collection device when collecting the video frame, and performing rendering processing on the mapped result to obtain an initial mask image; performing magnification processing on the initial mask image according to the size relationship between the transformed three-dimensional graph and the extracted three-dimensional graph to obtain the mask image.

5. The method of claim 2, wherein, In the three-dimensional graph and the mask image, different parts of the target object are displayed in different colors and / or different textures.

6. The method of claim 1, wherein, The feature extraction on the video frame and the mask image respectively to obtain first features and second features comprises: inputting the video frame and the mask map into a pre-trained end part segmentation model; performing feature extraction on the video frame through a first feature extraction network in the end part segmentation model to obtain first features, and performing feature extraction on the mask map through a second feature extraction network in the end part segmentation model to obtain second features.

7. The method of claim 6, wherein, The processing of the first features and the second features according to the connection relationship between parts in the target object respectively to obtain a first directed graph and a second directed graph comprises: processing the first features through a first graph neural network in the end part segmentation model to obtain a first directed graph, wherein each node in the first directed graph contains features of a part in the target object corresponding to the node, and the connection relationship between nodes in the first directed graph is determined according to the connection relationship between parts in the target object; and processing the second features through a second graph neural network in the end part segmentation model to obtain a second directed graph, wherein each node in the second directed graph contains features of a part in the target object corresponding to the node, and the connection relationship between nodes in the second directed graph is determined according to the connection relationship between parts in the target object.

8. The method of claim 7, wherein, The end part segmentation model learns the distribution and connection relationship of parts in the target object through graph network contrast learning during the training process; The loss function used by the end part segmentation model during the training process at least includes a contrast loss function for optimizing the first graph neural network and the second graph neural network; The optimization target of the contrast loss function includes that, in the first directed graph and the second directed graph obtained based on an input sample, the features of two nodes representing the same part in the target object are the same or similar, and the features of two nodes representing different parts in the target object are different.

9. The method of claim 7, wherein, The first graph neural network and the second graph neural network each contain at least two graph convolution network layers connected in sequence; The first graph convolution network layer in the first graph neural network is used to perform convolution operation on the features related to each part of the target object contained in the first features to obtain a convolution operation result, and the non-first graph convolution network layer in the first graph neural network is used to perform convolution operation again on the convolution operation result output by the previous graph convolution network layer; The first graph convolution network layer in the first graph neural network is used to perform convolution operation on the features related to each part of the target object contained in the first features to obtain a convolution operation result, and the non-first graph convolution network layer in the first graph neural network is used to perform convolution operation again on the convolution operation result output by the previous graph convolution network layer; 10. The method of claim 6, wherein, The first feature extraction network comprises a first encoder and a first transformer network; The first encoder is used to extract first local features from the video frame, and the first transformer network is used to extract first non-local features from the first local features output by the first encoder; The second feature extraction network comprises a second encoder and a second transformer network; The second encoder is configured to extract second local features from the mask image, and the second transformer network is configured to extract second non-local features from the second local features output by the second encoder.

11. The method of claim 10, wherein, The first directed graph is obtained by processing the first non-local features by a first graph neural network in the end part segmentation model; The second directed graph is obtained by processing the second non-local features by a second graph neural network in the end part segmentation model.

12. The method of claim 11, wherein, The first encoder and the second encoder are both convolutional neural network models, and the first transformer network and the second transformer network are both neural network models based on an attention mechanism.

13. The method of claim 10, wherein, The end part segmentation of the target object is obtained by performing segmentation processing on the target object according to the first feature, the second feature, the first directed graph and the second directed graph, comprising: The first non-local features, the second non-local features, the first directed graph and the second directed graph are spliced by a splicing layer in the end part segmentation model to obtain spliced features; The spliced features and the first local features are predicted by a decoder in the end part segmentation model to predict the end part segmentation result of the target object.

14. The method of claim 10, wherein, The end part segmentation model is obtained by supervised training based on input samples and labels thereof; the input samples comprise a video frame containing a target object and a mask image, and the mask image is rendered according to motion information synchronously acquired from the video frame; the labels comprise an end part annotation image of the target object in the video frame and annotation images of other parts; The loss function used by the end part segmentation model in the training process further comprises a first loss function and a second loss function; The optimization target of the first loss function comprises minimizing the difference between the predicted segmentation result output by the end part segmentation model in the training process and the end part annotation image; The optimization target of the second loss function comprises minimizing the difference between the predicted segmentation result of each part of the target object and the annotation image of each part of the target object; wherein the predicted segmentation result of each part of the target object is obtained based on the first non-local features output by the first transformer network.

15. The method according to any one of claims 1 to 14, characterized in that, The target object comprises at least one surgical instrument, the surgical instrument is held on a mechanical arm of a surgical robot, and a sensor is arranged on the mechanical arm, the sensor is configured to acquire motion information of the surgical instrument held on the mechanical arm.

16. The method of claim 15, wherein, The parts of the surgical instrument comprise a base, a wrist and an end execution part, the wrist is connected to the base and the end execution part respectively, so that the end execution part can move relative to the base; the end part of the target object is the end execution part.

17. An object segmentation apparatus characterized by comprising: Comprise: An image acquisition module is configured to acquire a video frame containing a target object and a mask image; the target object contains at least two parts moving relative to each other, and the mask image is rendered according to motion information of the target object collected synchronously with the video frame; A feature extraction module is configured to perform feature extraction on the video frame and the mask image respectively to obtain first features and second features; A directed graph acquisition module is configured to process the first features and the second features respectively according to a connection relationship between the parts in the target object to obtain a first directed graph and a second directed graph; the first directed graph and the second directed graph each include nodes corresponding to the parts in the target object respectively; A segmentation processing module is configured to perform segmentation processing on end parts of the target object according to the first features, the second features, the first directed graph and the second directed graph to obtain an end part segmentation result of the target object.

18. An electronic device, comprising: A computer program product includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to implement the method of any one of claims 1-16.

19. A computer-readable storage medium, characterized in that, A computer program product includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to implement the method of any one of claims 1-16.

Images