Training method, soft tissue deformation estimation method, device, equipment and storage medium

Abstract

The application provides a training method, a soft tissue deformation estimation method, a device, equipment and a storage medium. The training method comprises: obtaining a training sample set, different samples in the training sample set comprising point clouds at different time points and velocity vectors of three-dimensional points in the point clouds; wherein, any sample is obtained based on depth estimation and optical flow estimation of four soft tissue images collected from two different directions at adjacent time points; and a preset neural network model is self-supervised trained by using the samples in the training sample set, to obtain a soft tissue deformation estimation model. In the embodiment, the soft tissue deformation estimation model aggregates semantic information and motion flow information, and improves the accuracy of soft tissue deformation estimation.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to a training method for a soft tissue deformation estimation model, a soft tissue deformation estimation method, an apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Soft tissue deformation is ubiquitous during medical surgery. Estimating soft tissue deformation is fundamental to various downstream applications, including but not limited to intelligent sensing and surgical decision support (by estimating deformation, identifying and predicting soft tissue properties such as stiffness, and overlaying this information onto images for the surgeon to make intelligent decisions), and robotic surgical automation (by estimating soft tissue deformation, predicting and tracking key points on the tissue, which can then be used as feedback for visual servo control to achieve automated manipulation of the soft tissue). Despite extensive research on this task, the accuracy of soft tissue deformation estimation schemes in related technologies remains insufficient. Summary of the Invention

[0003] In view of this, this application provides a training method for a soft tissue deformation estimation model, a soft tissue deformation estimation method, an apparatus, an electronic device, and a computer-readable storage medium.

[0004] Specifically, this application is implemented through the following technical solution:

[0005] In a first aspect, embodiments of this application propose a training method for a soft tissue deformation estimation model, comprising:

[0006] A training sample set is obtained, wherein different samples in the training sample set include point clouds at different times and velocity vectors of three-dimensional points in the point clouds; wherein any sample is obtained by depth estimation and optical flow estimation based on four soft tissue images acquired from two different orientations at adjacent times.

[0007] The preset neural network model is trained in a self-supervised manner using samples from the training sample set to obtain a soft tissue deformation estimation model.

[0008] The self-supervised training instruction repeats the following iterative process until the iteration termination condition is met:

[0009] In each iteration, the velocity field output by the neural network model is obtained based on the samples in the training sample set, and the velocity field is integrated to obtain the first deformation field and the second deformation field that satisfy the differential homeomorphism mapping relationship.

[0010] Two soft tissue images acquired at the previous moment in the adjacent time interval are deformed using the first deformation field to obtain two first deformed images. Then, the two first deformed images are deformed using the second deformation field to obtain two second deformed images.

[0011] If the iteration termination condition is not met, the parameters of the neural network model are adjusted based on the four soft tissue images, the two first deformation images, and the two second deformation images to obtain the neural network model for the next iteration.

[0012] According to a second aspect of one or more embodiments of this specification, a method for estimating soft tissue deformation is proposed, comprising:

[0013] Acquire four soft tissue images taken from two different locations at adjacent time points;

[0014] Depth estimation and optical flow estimation are performed on the four soft tissue images to obtain the point cloud and the velocity vector of the three-dimensional points in the point cloud at the previous time in adjacent time intervals.

[0015] The velocity field is obtained by inputting the point cloud and the velocity vectors of the three-dimensional points in the point cloud into the soft tissue deformation estimation model; wherein the soft tissue deformation estimation model is trained based on the method described in any one of the first aspects;

[0016] The velocity field is integrated to obtain soft tissue deformation information based on the integration result.

[0017] According to a third aspect of the present disclosure, a training apparatus for a soft tissue deformation estimation model is provided, comprising:

[0018] The training sample acquisition module is used to acquire a training sample set, wherein different samples in the training sample set include point clouds at different times and velocity vectors of three-dimensional points in the point clouds; wherein any sample is obtained by depth estimation and optical flow estimation based on four soft tissue images acquired from two different orientations at adjacent times.

[0019] The self-supervised training module is used to perform self-supervised training on a preset neural network model using samples from the training sample set to obtain a soft tissue deformation estimation model.

[0020] Specifically, the self-supervised training module is used to repeat the following iterative process until the iteration termination condition is met: In each iteration, the velocity field output by the neural network model is obtained based on the samples in the training sample set, and the velocity field is integrated to obtain a first deformation field and a second deformation field that satisfy the differential homeomorphism mapping relationship; the first deformation field is used to deform two soft tissue images acquired at the previous time in the adjacent time to obtain two first deformation images, and the second deformation field is used to deform the two first deformation images to obtain two second deformation images; if the iteration termination condition is not met, the parameters of the neural network model are adjusted according to the four soft tissue images, the two first deformation images and the two second deformation images to obtain the neural network model in the next iteration.

[0021] According to a fourth aspect of the present disclosure, a soft tissue deformation estimation apparatus is provided, comprising:

[0022] The image acquisition module is used to acquire four soft tissue images taken from two different locations at adjacent time points;

[0023] The image processing module is used to perform depth estimation and optical flow estimation on the four soft tissue images to obtain the point cloud of the previous time step and the velocity vector of the three-dimensional points in the point cloud in the adjacent time steps.

[0024] The deformation estimation module is used to process the velocity vectors of the point cloud and the three-dimensional points in the point cloud into the soft tissue deformation estimation model to obtain the velocity field; wherein, the soft tissue deformation estimation model is trained based on the method described in any one of the first aspects;

[0025] The soft tissue deformation information acquisition module is used to integrate the velocity field to obtain soft tissue deformation information based on the integration result.

[0026] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising:

[0027] processor;

[0028] Memory used to store processor-executable instructions;

[0029] Wherein, when the processor executes the executable instructions, it is used to implement the method described in the first aspect.

[0030] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps of any of the methods described above.

[0031] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0032] In this embodiment, soft tissue deformation estimation can be achieved using a soft tissue deformation estimation model. This model is obtained through self-supervised training based on point clouds at different times and the velocity vectors of three-dimensional points within the point clouds. The point clouds at each time point and the velocity vectors of the three-dimensional points within the point clouds are obtained by depth estimation and optical flow estimation based on four soft tissue images acquired from two different orientations at adjacent times. The point clouds represent the semantic information of the soft tissue, and the velocity vectors of the three-dimensional points within the point clouds represent the motion flow information of the soft tissue. This allows the soft tissue deformation estimation model to aggregate the semantic and motion flow information of the soft tissue, enabling the perception of soft tissue deformation patterns in three-dimensional space. Furthermore, the obtained first and second deformation fields satisfy the differential homeomorphic mapping relationship, ensuring that the estimated deformation field conforms to the physical laws in the soft tissue deformation process. This, in turn, helps to improve the accuracy of the estimated deformation field and achieve accurate prediction of soft tissue deformation in actual three-dimensional space.

[0033] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of a robotic surgical system illustrated in an exemplary embodiment of this application.

[0035] Figure 2 This is a schematic diagram of a patient-side robot shown in an exemplary embodiment of this application.

[0036] Figure 3 This is a schematic flowchart illustrating a training method for a soft tissue deformation estimation model, as shown in an exemplary embodiment of this application.

[0037] Figure 4 This is a schematic diagram illustrating an exemplary embodiment of this application of acquiring a point cloud and the velocity vectors of three-dimensional points in the point cloud.

[0038] Figure 5 This is a schematic diagram illustrating the structure of a soft tissue deformation estimation model and the first and second deformation fields that satisfy the differential homeomorphic mapping relationship, as shown in an exemplary embodiment of this application.

[0039] Figure 6 This is a flowchart illustrating a soft tissue deformation estimation method according to an exemplary embodiment of this application.

[0040] Figure 7 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of this application.

[0041] Figure 8This is a schematic diagram of the structure of a training device for a soft tissue deformation estimation model, as illustrated in an exemplary embodiment of this application.

[0042] Figure 9 This is a schematic diagram of the structure of a soft tissue deformation estimation device shown in an exemplary embodiment of this application. Detailed Implementation

[0043] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0044] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0045] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0046] Soft tissue refers to various non-skeletal tissues, including muscles, fat, skin, blood vessels, nerves, and internal organs. These soft tissues play vital roles in supporting, protecting, and performing functions within animals. Soft tissue deformation is ubiquitous during surgical procedures. Estimating soft tissue deformation is fundamental to various downstream applications and is a prerequisite for many downstream tasks or applications, including intraoperative navigation, tool-tissue interaction analysis, surgical assessment, anatomical biomechanical estimation, and automated soft tissue manipulation.

[0047] For example, intelligent perception and surgical decision support: by estimating soft tissue deformation, the system can identify and predict attributes such as the hardness of the soft tissue, and this relevant information can be overlaid on the image and presented to the doctor, providing intelligent decision-making for the doctor's surgery.

[0048] For example, in robotic surgery automation: by estimating the deformation of soft tissue, key points on the tissue can be predicted and tracked. This information can be used as feedback for visual servo control to achieve automated manipulation of soft tissue.

[0049] The difficulty in estimating soft tissue deformation lies in: (1) the environment of soft tissue operation is highly complex and dynamic; (2) the obstruction of surgical instruments causes the soft tissue to be partially visible during the deformation process; and (3) the visual field is blurred due to tissue movement, lighting changes, smoke and blood during the operation.

[0050] One approach in related technologies estimates soft tissue deformation by tracking and matching pixel-level dense feature points. However, this method has several drawbacks in surgical scenarios. First, the background and texture in surgical scenarios are usually quite uniform, and the monotonous texture and background lead to inaccurate point-to-point matching. Second, feature point matching is based on two-dimensional images and lacks perception of three-dimensional structures. Some matched feature points that appear reasonable in two-dimensional space may not be reasonable when projected into three-dimensional space. This results in inaccurate estimations of soft tissue deformation.

[0051] Based on this, this application provides a soft tissue deformation estimation method, which can realize soft tissue deformation estimation by means of a soft tissue deformation estimation model. The soft tissue deformation estimation model is obtained by self-supervised training based on point clouds at different times and the velocity vectors of three-dimensional points in the point clouds. The point clouds at each time and the velocity vectors of three-dimensional points in the point clouds are obtained by depth estimation and optical flow estimation based on four soft tissue images acquired from two different orientations at adjacent times. The point clouds represent the semantic information of the soft tissue, and the velocity vectors of three-dimensional points in the point clouds represent the motion flow information of the soft tissue. This allows the soft tissue deformation estimation model to aggregate the semantic information and motion flow information of the soft tissue, realize the perception of the soft tissue deformation law from three-dimensional space, and the obtained first deformation field and second deformation field satisfy the differential homeomorphic mapping relationship, ensuring that the estimated deformation field conforms to the physical law in the soft tissue deformation process, thereby improving the accuracy of the estimated deformation field and realizing accurate estimation of soft tissue deformation in actual three-dimensional space.

[0052] For example, the soft tissue deformation estimation method and the training method of the soft tissue deformation estimation model can be executed by electronic devices, including but not limited to servers, cloud servers, smartphones / mobile phones, tablet computers, personal digital assistants (PDAs), laptop computers, desktop computers, media content players, video game consoles / systems, virtual reality systems, augmented reality systems, wearable devices (e.g., watches, glasses, gloves, headwear (e.g., hats, helmets, virtual reality headsets, augmented reality headsets, head-mounted devices (HMDs), headbands)) or any other type of device. It is understood that the soft tissue deformation estimation method and the training method of the soft tissue deformation estimation model can be executed by different electronic devices or by the same electronic device; this embodiment does not impose any limitations on this.

[0053] In an exemplary application scenario, minimally invasive surgery based on surgical robots offers advantages such as accurate positioning, stable operation, minimal surgical trauma, and rapid postoperative recovery. Computer-aided surgical robot automation, as a crucial component of smart healthcare, can reduce the rate of misjudgment by surgeons during surgery and decrease the tedious and repetitive surgical tasks associated with fixed procedures. Among these tasks, soft tissue deformation estimation plays a vital role in robot-based minimally invasive surgery. By estimating the deformation of soft tissue, the surgical robot can predict and track key points on the tissue. This information can then serve as feedback for visual servo control, enabling automated manipulation of the soft tissue.

[0054] For example, please refer to Figure 1 This is a schematic diagram of a robotic surgical system 100. During surgery, the patient is positioned in front of a patient-side robot (PSR) 101, which includes one or more robotic arms 101a. The end of each robotic arm 101a is used to hold one or more surgical instruments 001. The surgeon can control the robotic arms 101a through a surgeon console (SGC) 102, thereby controlling the surgical instruments 001 to perform surgical operations on the patient. The robotic arms 101a can also hold image acquisition devices (e.g., endoscopes, not shown). The surgeon can control the robotic arm 101a holding the endoscope through the surgeon console 102, thereby moving and holding the endoscope near the patient's lesion area to acquire surgical images including the patient's lesion and surrounding tissues, and the surgical instruments 001, i.e., soft tissue images mentioned in the embodiments of this application.

[0055] During the surgery, the surgical instruments 001 and / or endoscope on the robotic arm 101a are inserted into the patient's body through a pre-set incision and can rotate around the center point of that incision (generally referred to as the Remote Center of Motion point, RCM). The images acquired by the endoscope are transmitted to the Vision Cart (VCT) 103 for image processing and recording, and the processed images are displayed on the respective display devices of the Vision Cart 103 and the doctor's console 102 for observation by the doctor and other surgical staff.

[0056] For example, please refer to Figure 2 A schematic diagram of a patient-side robot 101 is shown. The patient-side robot 101 includes a chassis 101b, a push handle 101c, and at least one robotic arm 101a (only one robotic arm 101a is shown in the diagram for illustration purposes). Each robotic arm 101a includes an adjustment arm 101a-1 and an operating arm 101a-2. The robotic arm 101a-2 may include an image acquisition device, which can acquire soft tissue images near the patient's lesion area. This allows for the training of a soft tissue deformation estimation model based on the acquired soft tissue images, as well as subsequent soft tissue deformation estimation processes.

[0057] In some embodiments, the training process of a soft tissue deformation estimation model is illustrated by way of example:

[0058] The training process of a soft tissue deformation estimation model can be as follows: first, a model is represented by modeling; then, an evaluation function is constructed to evaluate the model; finally, the evaluation function is optimized based on sample data and optimization methods to adjust the model to its optimal state.

[0059] 1. Modeling: During training, the first step is to select a model architecture suitable for the task. The model can be a neural network, decision tree, support vector machine, etc. The goal of modeling is to select a model that can effectively fit the data and has appropriate complexity.

[0060] 2. Evaluation Function: During training, an evaluation function needs to be defined to measure the difference or error between the model's predictions and the true labels. This evaluation function is usually called the loss function, and it measures the model's performance.

[0061] 3. Optimization Method: During training, an optimization method needs to be selected to minimize the loss function. Common optimization methods include Stochastic Gradient Descent (SGD), Adam, and RMSprop. These methods improve the model's predictive ability by adjusting the model parameters to gradually reduce the loss function.

[0062] 4. Training Data: Model training requires a large amount of training data. This data is typically divided into a training set, a validation set, and a test set. The training set is used for parameter updates, the validation set is used for tuning model hyperparameters, and the test set is used to evaluate the model's generalization ability on unseen data.

[0063] 5. Training Iterations: During model training, model parameters are gradually adjusted through iterative optimization. Each training iteration includes steps such as forward propagation, calculating the loss function value, backpropagation, and parameter updates. The number of training iterations is usually set to a fixed number or dynamically adjusted based on the model's performance on the validation set.

[0064] Through the above steps, the model can be trained using sample data and, after multiple iterations of optimization, ultimately yield an optimal model capable of accurate prediction and generalization. Note that during model training, techniques such as overfitting control, learning rate adjustment, and regularization are also necessary to improve the model's training effectiveness and generalization ability.

[0065] In some embodiments, please refer to Figure 3 , Figure 3 A flowchart illustrating a training method for a soft tissue deformation estimation model is shown. The method can be executed by an electronic device and includes:

[0066] In S101, a training sample set is obtained, wherein different samples in the training sample set include point clouds at different times and velocity vectors of three-dimensional points in the point clouds; wherein any sample is obtained by depth estimation and optical flow estimation based on four soft tissue images acquired from two different orientations at adjacent times.

[0067] For example, the point cloud in any sample is obtained by depth estimation based on four soft tissue images acquired from two different orientations at adjacent time points, and the velocity vector of the three-dimensional points in the point cloud of any sample is obtained by optical flow estimation based on four soft tissue images acquired from two different orientations at adjacent time points.

[0068] For example, the four soft tissue images corresponding to any sample include the left and right views acquired at the previous time step in an adjacent time step, and the left and right views acquired at the next time step in an adjacent time step. Of course, in addition to the two different orientations of the left and right views, there can be other different orientations, and this embodiment does not impose any restrictions on this.

[0069] In S102, the preset neural network model is self-supervised and trained using samples from the training sample set to obtain a soft tissue deformation estimation model.

[0070] The self-supervised training instruction repeats the following iterative process until the iteration termination condition is met: In each iteration, the velocity field output by the neural network model is obtained based on the samples in the training sample set; the velocity field is integrated to obtain a first deformation field and a second deformation field that satisfy the differential homeomorphism mapping relationship; the first deformation field is used to deform two soft tissue images acquired at the previous time step in the adjacent time steps to obtain two first deformation images; the second deformation field is then used to deform the two first deformation images to obtain two second deformation images; if the iteration termination condition is not met, the parameters of the neural network model are adjusted according to the four soft tissue images, the two first deformation images, and the two second deformation images to obtain the neural network model in the next iteration.

[0071] In this embodiment, each sample in the training sample set includes a point cloud representing the semantic information of soft tissue and a velocity vector representing the motion flow information of soft tissue. Then, based on the samples in the training sample set, a preset neural network model is trained in a self-supervised manner to obtain a soft tissue deformation estimation model that aggregates the semantic information and motion flow information of soft tissue. The soft tissue deformation estimation model estimates the stable velocity field of soft tissue. Then, by integrating the velocity field, the first deformation field and the second deformation field that satisfy the differential homeomorphism mapping relationship are obtained, ensuring that the estimated deformation field conforms to the physical laws in the process of soft tissue deformation, thereby improving the accuracy of the estimated deformation field and enabling the trained soft tissue deformation estimation model to accurately predict the deformation of soft tissue in actual three-dimensional space.

[0072] I. Regarding training samples.

[0073] In some embodiments, considering that surgical instruments may obscure the soft tissue in the acquired soft tissue images, a pre-trained instrument segmentation network can be used to improve the accuracy of soft tissue deformation estimation. Figure 1 and Figure 2 The surgical instruments in the soft tissue image acquired by the image acquisition device are segmented to obtain a soft tissue image with the surgical instruments removed.

[0074] For example, soft tissue images acquired by an image acquisition device and masks of surgical instruments can be input into a pre-trained instrument segmentation network. The surgical instruments comprise at least two parts that move relative to each other. The masks are rendered based on motion information of the surgical instruments acquired synchronously with the soft tissue images. For example... Figure 2 The robotic arm 101a-2 may include one or more sensors, such as displacement sensors, orientation sensors, and / or position sensors. The motion information of the robotic arm 101a and the surgical instrument 001 held on the robotic arm 101a can be obtained through the detection values ​​of these sensors, such as the pose information of various parts of the surgical instrument 001.

[0075] The instrument segmentation network extracts features from the soft tissue image and the mask image, obtaining a first feature and a second feature. Then, based on the connectivity between different parts of the surgical instrument, the first and second features are processed to obtain a first directed graph and a second directed graph. Each of the first and second directed graphs includes nodes corresponding to different parts of the surgical instrument. Finally, the distal end of the surgical instrument is segmented based on the first feature, the second feature, the first directed graph, and the second directed graph, yielding the segmentation result. The electronic device can then process the soft tissue image acquired by the image acquisition device based on the segmentation result of the distal end of the surgical instrument, thereby obtaining a soft tissue image without the surgical instrument.

[0076] In some embodiments, the point cloud contained in any sample is illustrated exemplarily here: After acquiring four soft tissue images from two different orientations at adjacent time points, depth estimation can be performed on two soft tissue images acquired from two different orientations at the previous time point in the adjacent time points to obtain first depth information; using the first depth information and the intrinsic and extrinsic parameters of the image acquisition device, the pixels in the soft tissue image acquired from one orientation at the previous time point in the adjacent time points are mapped from two-dimensional space to three-dimensional space to obtain the point cloud of the previous time point in the adjacent time points. This embodiment realizes the acquisition of semantic information of three-dimensional space from two-dimensional soft tissue images. The point cloud can provide information on where the soft tissue deformation occurs in three-dimensional space, which is beneficial for accurately estimating the deformation of the soft tissue in three-dimensional space (i.e., real-world space) rather than in two-dimensional space.

[0077] In some embodiments, the velocity vector of a 3D point in a point cloud contained in any sample is illustrated by way of example: After acquiring four soft tissue images from two different orientations at adjacent time points, depth estimation can be performed on two soft tissue images from two different orientations at the previous time point to obtain first depth information; and depth estimation can be performed on two soft tissue images from two different orientations at the next time point to obtain second depth information; then, optical flow estimation is performed on two soft tissue images from the same orientation at adjacent time points using an optical flow estimation network to obtain the velocity vector of the pixel; finally, the velocity vector of the pixel is projected and mapped using the first depth information and the second depth information to obtain the velocity vector of the 3D point in the point cloud. This embodiment achieves the acquisition of velocity vectors in 3D space, which can inform the direction and distance of the soft tissue displacement in 3D space, thereby facilitating the accurate estimation of the deformation of the soft tissue in 3D space (i.e., real-world space) rather than in 2D space.

[0078] For example, a depth estimation model can be pre-trained for depth estimation. For instance, the STTR (STereo Transformer) depth estimation model can be trained, which predicts the depth information of each pixel from a soft tissue image based on a Transformer architecture. By introducing self-attention and Transformer attention mechanisms, the STTR model can automatically learn features and relationships from deformed images and generate pixel-level depth estimates. The STTR model includes an encoder and a decoder. The encoder extracts feature representations from the input deformed icon, while the decoder uses these features to predict the depth pixel-by-pixel. The STTR model is trained by minimizing the difference between the predicted and true depths and updates the network parameters using backpropagation. Through self-attention and Transformer attention mechanisms, the STTR model can capture the contextual relationships between pixels, improving the accuracy of depth estimation.

[0079] Two soft tissue images acquired from two different orientations at an adjacent time step can be input into the STTR model for processing to obtain the first depth information output by the STTR model. Similarly, two soft tissue images acquired from two different orientations at an adjacent time step can be input into the STTR model for processing to obtain the second depth information output by the STTR model.

[0080] For example, for an optical flow estimation network, a self-supervised loss consisting of an occlusion-aware photometric consistency term and an edge-aware smoothing term can be used to train the optical flow estimation network. The occlusion-aware photometric consistency term represents the difference between the predictions of the optical flow estimation network and the true labels; the edge-aware smoothing term is a first-order edge-aware smoothness, the goal of which is to smooth the predictions of the optical flow estimation network.

[0081] For example, please refer to Figure 4 , This represents the left view at time t, which is an adjacent time interval. This represents the right view at time t, which is an adjacent time interval. This represents the left view at time t+1, which is an adjacent time interval. This represents the right view of time t+1 within a given time interval. Electronic devices can... and Depth estimation is performed to obtain the first depth information D. t That is, the depth map corresponding to the left view at time t; and the depth map corresponding to the left view at time t. and Depth estimation is performed to obtain the second depth information D. t+1 That is, the depth map corresponding to the left view at time t+1. Electronic devices can also utilize pre-trained instrument segmentation networks to... and The surgical instruments in the image are segmented to obtain the left view M at time t, after the surgical instruments have been removed. t Left view M at time t+1 t+1 Electronic devices can be based on D. t M t By back-projecting the intrinsic and extrinsic parameters of the image acquisition device, the point cloud S at time t is obtained. t Electronic devices can... t and M t+1 The input optical flow estimation network performs optical flow estimation processing to obtain the pixel velocity vector; then, D... t and D t+1 By projecting and mapping the velocity vectors of pixels, the velocity vectors V of the three-dimensional points in the point cloud are obtained. t .

[0082] II. Design of Model Structure and Loss Function.

[0083] In some embodiments, the electronic device can use samples from the training sample set to perform self-supervised training on a preset neural network model to obtain a soft tissue deformation estimation model.

[0084] Please see Figure 5The neural network model (or soft tissue deformation estimation model) includes a first encoder, a second encoder, and a decoder. The first encoder extracts features from the point cloud contained in the sample, obtaining point cloud features. The second encoder extracts features from the velocity vectors of the three-dimensional points in the point cloud contained in the sample, obtaining motion features. The decoder performs deformation estimation based on the point cloud features and motion features to obtain the velocity field.

[0085] For example, a neural network model includes the U-shaped neural network model. The U-shaped neural network model is a convolutional neural network (CNN) architecture commonly used for semantic segmentation tasks. It consists of an encoder and a decoder. The encoder is primarily responsible for feature extraction, while the decoder uses upsampling and skip connections to restore the feature maps extracted by the encoder to an output with the same size as the input image. In the U-shaped neural network model, the encoder typically consists of multiple convolutional and pooling layers stacked alternately to extract image features layer by layer while reducing the image size. This reduces computation and memory consumption while preserving important features, improving training speed and performance. The decoder uses upsampling or deconvolution operations to restore the low-resolution feature maps generated by the encoder to a high-resolution output with the same size as the original input image. Furthermore, to overcome problems such as information loss and blurring, the U-shaped neural network model introduces skip connections, allowing the decoder to skip some layers from the encoder and directly use lower-level feature maps for fine-grained modeling. Due to its excellent feature extraction and segmentation performance, the U-shaped neural network model has been widely used in image segmentation, medical image analysis, natural language processing and other fields, and has become one of the mainstream models in current semantic segmentation tasks.

[0086] The neural network model provided in this application is based on a U-shaped neural network model, including a first encoder, a second encoder, and a decoder. The first encoder and second encoder are connected in a skip connection, as are the second encoder and decoder. Please refer to... Figure 5 , Figure 5 The dashed line between the first encoder and the second encoder in the diagram represents a skip connection, and the dashed line between the second encoder and the decoder also represents a skip connection.

[0087] For example, both the first encoder and the second encoder contain N neural network layers; N is an integer greater than 1; the neural network layers are convolutional layers or pooling layers. The neural network layers in the first encoder and the second encoder are connected in a skip connection; the input data of the (n+1)th neural network layer in the first encoder is the output data of the nth neural network layer in the first encoder; the input data of the (n+1)th neural network layer in the second encoder is the result of concatenating the output data of the nth neural network layer in the second encoder and the output data of the nth neural network layer in the first encoder, where 1 < n ≤ N. In this embodiment, the skip connection between the first encoder and the second encoder enables the output of the second encoder to fuse point cloud features and motion features.

[0088] The decoder comprises N neural network layers; N is an integer greater than 1. The neural network layers in the decoder are connected in skip connections to the neural network layers in the second encoder. The input data of the first neural network layer in the decoder includes the output data of the last neural network layer in the second encoder. It is understood that, since the inputs of non-first neural network layers in the second encoder are all fused with data from the outputs of the neural network layers in the first encoder, the output data of the last neural network layer in the second encoder includes point cloud features and motion features. The input data of the (m+1)th neural network layer in the decoder is the result of concatenating the output data of the mth neural network layer in the decoder with the output data of the neural network layer corresponding to the (m+1)th neural network layer in the decoder, where 1 < m ≤ N. The output of the last neural network layer in the decoder is the velocity field of the previous time step in an adjacent time step. In this embodiment, the decoder can skip some layers from the second encoder and directly use low-level features for fine modeling, thereby overcoming problems such as information loss and fuzziness.

[0089] For example, to improve the accuracy of soft tissue estimation, please refer to Figure 5 The input data of the last neural network layer in the decoder includes not only the output data of the previous neural network layer in the decoder and the output data of the neural network layer in the second encoder corresponding to the last neural network layer in the decoder, but also the velocity field estimated by the neural network model at the previous moment. In this embodiment, using the velocity field estimated by the neural network model at the previous moment as one of the reference information for predicting the velocity field at the current moment enables better temporal modeling and better ensures that the subsequently predicted deformation field has temporal consistency. In other words, the neural network model (i.e., the soft tissue deformation estimation model) provided in this application embodiment can aggregate semantic information, motion flow information, and long-range temporal context, making it both temporally consistent and physically plausible.

[0090] Understandably, if there is no velocity field from the previous moment, then a velocity field with all zeros is used as input.

[0091] In some embodiments, the self-supervised training process is an iterative optimization process. During self-supervised training, the following iterative process can be repeated until the iteration termination condition is met: In each iteration, the velocity field output by the neural network model is obtained based on samples in the training sample set. The velocity field is integrated to obtain a first deformation field and a second deformation field that satisfy the differential homeomorphism mapping relationship. Then, the electronic device can use the first deformation field to deform two soft tissue images acquired at the previous time step in adjacent time steps to obtain two first deformation images. Then, the second deformation field is used to deform the two first deformation images to obtain two second deformation images. If the iteration termination condition is not met, the parameters of the neural network model are adjusted based on four soft tissue images, two first deformation images, and two second deformation images to obtain the neural network model for the next iteration.

[0092] For example, the iteration termination conditions include, but are not limited to: reaching a preset number of iterations, the difference between the first deformed image and the soft tissue image acquired at the next adjacent time step being less than a first preset difference, and / or the difference between the second deformed image and the soft tissue image acquired at the previous adjacent time step being less than a second preset difference. The preset number of iterations, the first preset difference, and the second preset difference can be specifically set according to the actual application scenario, and this embodiment does not impose any restrictions on them.

[0093] Please see Figure 5 In obtaining the velocity field v output by the neural network model t The electronic device can then integrate the velocity field. This integration process can be approximated using a scaling and squaring algorithm to obtain the first deformation field φ that satisfies the differential homeomorphism mapping relation. t→t+1 Second deformation field φ t+1→t Using differential homeomorphism modeling better guarantees that the obtained deformation field is spatially smooth and reversible bijective. The first deformation field φ t→t+1 The second deformation field φ characterizes the deformation from one time point (t) to the next (t+1) in an adjacent time interval. t+1→t It represents the deformation from the next time step (t+1) to the previous time step (t) in an adjacent time interval.

[0094] Since the samples in the training sample set are three-dimensional information, the velocity field output by the neural network model is also three-dimensional. The first and second deformation fields obtained by integrating the velocity field and satisfying the differential homeomorphism mapping relationship are also three-dimensional. Therefore, the first and second deformation images are generated by projecting the deformed point cloud into two-dimensional space.

[0095] The generation process of the first deformed image is explained here: The above-mentioned deformation processing of two soft tissue images acquired at the previous moment in adjacent time intervals using the first deformation field to obtain two first deformed images includes: for each soft tissue image acquired at the previous moment in adjacent time intervals, the point cloud corresponding to that soft tissue image is deformed using the first deformation field to obtain a first deformed point cloud; then, using the intrinsic and extrinsic parameters of the image acquisition device, the first deformed point cloud is projected onto a two-dimensional space to obtain the first deformed image corresponding to that soft tissue image. The point cloud corresponding to that soft tissue image can be obtained by mapping the soft tissue image from two-dimensional space to three-dimensional space based on the aforementioned first depth information and the intrinsic and extrinsic parameters of the image acquisition device.

[0096] For example, for the left view at time t, the predicted left view at time t+1 (i.e., the first deformed image) can be obtained based on the above steps. Similarly, for the right view at time t, the predicted right view at time t+1 (i.e., the first deformed image) can be obtained based on the above steps.

[0097] The process of generating the second deformed image is explained here: The above-mentioned deformation processing of two first deformed images using the second deformation field to obtain two second deformed images includes: for each first deformed image, the first deformed point cloud corresponding to the first deformed image is deformed again using the second deformation field to obtain the second deformed point cloud; then, using the internal and external parameters of the image acquisition device, the second deformed point cloud is projected onto a two-dimensional space to obtain the second deformed image.

[0098] Using the previous example, for the predicted left view at time t+1, the predicted left view at time t (i.e., the second deformed image) can be obtained based on the above steps. Similarly, for the predicted right view at time t+1, the predicted right view at time t (i.e., the second deformed image) can be obtained based on the above steps.

[0099] For example, this application uses cycle consistency as the learning objective of self-supervised training, which causes the soft tissue image to return to its original state after undergoing a forward-backward cycle, and the state during the deformation is consistent with the state presented in the real image at that time. Specifically, the learning objectives of self-supervised training include: (1) the first deformed image is the same as or similar to the soft tissue image acquired at the next adjacent time, and (2) the second deformed image is the same as or similar to the soft tissue image acquired at the previous adjacent time. The first learning objective ensures that the estimation result of soft tissue deformation is consistent with the actual deformation of soft tissue; the second learning objective makes the soft tissue deformation estimation model robust to discontinuous semantic sequences caused by surgical instrument occlusion. Combining these two learning objectives helps to generate a stronger learning signal and avoids obtaining trivial solutions through optimization.

[0100] Based on the first learning objective mentioned above, a first loss function can be constructed; and based on the second learning objective mentioned above, a second loss function can be constructed.

[0101] For example, an electronic device can calculate a first loss value corresponding to a first loss function based on two first deformation images and two soft tissue images acquired at a later time in an adjacent time period; and calculate a second loss value corresponding to a second loss function based on two second deformation images and two soft tissue images acquired at an earlier time in an adjacent time period; and then adjust the parameters of the neural network model based on the first and second loss values. For example, the parameters of the neural network model can be adjusted based on the weighted sum of the first and second loss values.

[0102] The first loss function is used to calculate the photometric loss between the pixel values ​​of the soft tissue image acquired at the next adjacent time and the pixel values ​​of the first deformed image belonging to the same orientation; then the first loss value includes the photometric loss value between the pixel values ​​of the soft tissue image acquired at the next adjacent time and the pixel values ​​of the first deformed image belonging to the same orientation.

[0103] Using the previous example, the photometric loss between the pixel value of the pixel in the left view at time t+1 and the pixel value of the pixel in the predicted left view (first deformed image) at time t+1 can be calculated using the first loss function to obtain the photometric loss value; and the photometric loss between the pixel value of the pixel in the right view at time t+1 and the pixel value of the pixel in the predicted right view (first deformed image) at time t+1 can also be calculated using the first loss function to obtain the photometric loss value; then the two photometric loss values ​​are weighted and summed to obtain the first loss value mentioned above.

[0104] For two deformed images, the same loss function can be used to calculate the photometric loss; or different loss functions can be used to calculate the photometric loss (that is, the first loss function includes two different sub-functions for calculating the photometric loss).

[0105] To illustrate, let's consider two deformed images using two different sub-functions to calculate the photometric loss: For the left view, we can perform Census transforms on both the left view at time t+1 and the predicted left view (the first deformed image) at time t+1. The Hamming distance between the two transformed images is then calculated as the photometric loss value. The Census transform, a non-parametric image transform, effectively detects local structural features in an image, such as edges and corners. Essentially, the Census transform encodes the grayscale values ​​of image pixels into a binary code stream to obtain the relationship between the grayscale values ​​of neighboring pixels and the grayscale value of the center pixel. For the right view, we can calculate the angle between the RGB vectors of the pixels in the right view at time t+1 and the RGB vectors of the pixels in the predicted right view (the first deformed image) at time t+1 to obtain the photometric loss value.

[0106] The second loss function is used to calculate the L1 norm distance between the pixel coordinates in the soft tissue image acquired at the previous time step in adjacent time steps and the pixel coordinates in the second deformation image belonging to the same orientation; then the second loss value includes the L1 norm distance between the pixel coordinates in the soft tissue image acquired at the previous time step in adjacent time steps and the pixel coordinates in the second deformation image belonging to the same orientation.

[0107] Continuing with the previous example, the L1 norm distance between the pixel coordinates of a pixel in the left view at time t and the pixel coordinates of a pixel in the predicted left view (i.e., the second deformed image) at time t can be calculated using the second loss function. Similarly, the L1 norm distance between the pixel coordinates of a pixel in the right view at time t and the pixel coordinates of a pixel in the predicted right view (i.e., the second deformed image) at time t can be calculated using the second loss function. The two L1 norm distances are then weighted and summed to obtain the aforementioned second loss value; alternatively, one of the L1 norm distances can be selected as the aforementioned second loss value. This embodiment does not impose any restrictions on this.

[0108] In some embodiments, to accelerate model convergence, a third loss function can be added. This third loss function determines the similarity between the velocity vectors of the 3D points in the point cloud and the first deformation field; for example, the L1 norm distance between the velocity vectors of the 3D points in the point cloud and the first deformation field can be calculated as the third loss value. The optimization objective of the third loss function is to minimize the L1 norm distance. That is, the electronic device can calculate the third loss value corresponding to the third loss function based on the similarity between the velocity vectors of the 3D points in the point cloud and the first deformation field; and then adjust the parameters of the neural network model based on the third loss value, the first loss value, and the second loss value.

[0109] In some embodiments, to avoid abrupt changes in the deformation results, a fourth loss function can be added. This fourth loss function smooths out the first and second deformation fields. That is, the electronic device can calculate the fourth loss value corresponding to the fourth loss function based on the gradient change between the first and second deformation fields; then, it adjusts the parameters of the neural network model based on the fourth loss value, the first loss value, and the second loss value. For example, the fourth loss function could be a first-order edge-aware smoothness function, whose optimization objective is to make the deformation field output smooth, avoiding abrupt deformation results.

[0110] In some embodiments, the parameters of the neural network model can be adjusted by combining the third and fourth loss values, the first loss value, and the second loss value, thereby improving the model training efficiency and the accuracy of the output results.

[0111] The following is an example illustrating the application process of the soft tissue deformation estimation model:

[0112] Please see Figure 6 This application provides a method for estimating soft tissue deformation, including:

[0113] In S201, four soft tissue images acquired from two different orientations at adjacent time points are obtained.

[0114] In S202, depth estimation and optical flow estimation are performed on the four soft tissue images to obtain the point cloud of the previous time step and the velocity vector of the three-dimensional points in the point cloud in adjacent time steps.

[0115] In S203, the velocity vectors of the point cloud and the three-dimensional points in the point cloud are input into the soft tissue deformation estimation model for processing to obtain the velocity field; wherein, the soft tissue deformation estimation model is trained based on the training method of the soft tissue deformation estimation model described above.

[0116] In S204, the velocity field is integrated to obtain soft tissue deformation information based on the integration result.

[0117] In this embodiment, point clouds representing the semantic information of soft tissue and velocity vectors representing the motion flow information of soft tissue are input into the soft tissue deformation estimation model. The soft tissue deformation estimation model aggregates the semantic information and motion flow information of soft tissue, enabling the perception of soft tissue deformation patterns in three-dimensional space. Furthermore, the velocity field is integrated to obtain an integral result that satisfies the differential homeomorphism mapping relationship, ensuring that the estimated soft tissue deformation information conforms to the physical laws of the soft tissue deformation process, thereby achieving accurate prediction of soft tissue deformation in actual three-dimensional space.

[0118] In some embodiments, the four soft tissue images include left and right views acquired at the previous time step in adjacent time steps, and left and right views acquired at the next time step in adjacent time steps.

[0119] In some embodiments, obtaining the point cloud of the previous moment in an adjacent time period includes: performing depth estimation on two soft tissue images acquired from two different orientations at the previous moment in the adjacent time period to obtain first depth information; and using the first depth information and the intrinsic and extrinsic parameters of the image acquisition device (camera parameters) to map the pixels in the soft tissue image acquired from one orientation of the target at the previous moment in the adjacent time period from two-dimensional space to three-dimensional space to obtain the point cloud of the previous moment in the adjacent time period.

[0120] In some embodiments, obtaining the velocity vector of a 3D point in a point cloud includes: performing depth estimation on two soft tissue images acquired from two different orientations at the previous time in the adjacent time to obtain first depth information; and performing depth estimation on two soft tissue images acquired from two different orientations at the next time in the adjacent time to obtain second depth information; performing optical flow estimation on two soft tissue images acquired from the same orientation at the adjacent time using an optical flow estimation network to obtain the velocity vector of a pixel; and projecting the velocity vector of the pixel using the first depth information and the second depth information to obtain the velocity vector of the 3D point in the point cloud.

[0121] In some embodiments, the method further includes: obtaining the velocity field estimated by the soft tissue deformation estimation model at the previous moment. The step of inputting the point cloud and the velocity vectors of the three-dimensional points in the point cloud into the soft tissue deformation estimation model for processing to obtain the velocity field includes: inputting the point cloud, the velocity vectors of the three-dimensional points in the point cloud, and the velocity field estimated by the soft tissue deformation estimation model at the previous moment into the soft tissue deformation estimation model for processing to obtain the velocity field. In this embodiment, the soft tissue deformation estimation model can aggregate semantic information, motion flow information, and long-range temporal context, making the output result both temporally consistent and physically plausible.

[0122] The various technical features in the above embodiments can be combined arbitrarily, as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of various technical features in the above embodiments is also within the scope of this specification.

[0123] Corresponding to the aforementioned embodiments of the training method for the soft tissue deformation estimation model, this application also provides embodiments of a training apparatus for the soft tissue deformation estimation model.

[0124] The training device for the soft tissue deformation estimation model of this application can be applied to electronic devices. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of the electronic device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 7 The diagram shown is a hardware structure diagram of an electronic device used for training the soft tissue deformation estimation model of this application, except for... Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in the embodiment may also include other hardware depending on the actual function of the training device for the soft tissue deformation estimation model, which will not be described in detail here.

[0125] Please refer to Figure 8 This application provides a training apparatus for a soft tissue deformation estimation model, comprising:

[0126] The training sample acquisition module 301 is used to acquire a training sample set, wherein different samples in the training sample set include point clouds at different times and velocity vectors of three-dimensional points in the point clouds; wherein any sample is obtained by depth estimation and optical flow estimation based on four soft tissue images acquired from two different orientations at adjacent times.

[0127] The self-supervised training module 302 is used to perform self-supervised training on a preset neural network model using samples from the training sample set to obtain a soft tissue deformation estimation model.

[0128] Specifically, the self-supervised training module is used to repeat the following iterative process until the iteration termination condition is met: In each iteration, the velocity field output by the neural network model is obtained based on the samples in the training sample set, and the velocity field is integrated to obtain a first deformation field and a second deformation field that satisfy the differential homeomorphism mapping relationship; the first deformation field is used to deform two soft tissue images acquired at the previous time in the adjacent time to obtain two first deformation images, and the second deformation field is used to deform the two first deformation images to obtain two second deformation images; if the iteration termination condition is not met, the parameters of the neural network model are adjusted according to the four soft tissue images, the two first deformation images and the two second deformation images to obtain the neural network model in the next iteration.

[0129] In some embodiments, the first deformation field characterizes the deformation from the previous moment to the next moment in the adjacent time intervals, and the second deformation field characterizes the deformation from the next moment to the previous moment in the adjacent time intervals.

[0130] In some embodiments, the learning objectives of the self-supervised training include: the first deformed image being the same as or similar to the soft tissue image acquired at the next adjacent time step, and the second deformed image being the same as or similar to the soft tissue image acquired at the previous adjacent time step.

[0131] In some embodiments, the iteration termination conditions include: reaching a preset number of iterations, the difference between the first deformed image and the soft tissue image acquired at the next adjacent time moment being less than a first preset difference, and / or the difference between the second deformed image and the soft tissue image acquired at the previous adjacent time moment being less than a second preset difference.

[0132] In some embodiments, the self-supervised training module 302 is specifically used to: calculate a first loss value based on two first deformation images and two soft tissue images acquired at the later time of the adjacent time; and calculate a second loss value based on two second deformation images and two soft tissue images acquired at the earlier time of the adjacent time; and adjust the parameters of the neural network model based on the first loss value and the second loss value.

[0133] In some embodiments, the first loss value includes the photometric loss value between the pixel value of the soft tissue image acquired at the later time in the adjacent time and the pixel value of the first deformation image belonging to the same orientation; the second loss value includes the L1 norm distance between the pixel coordinates in the soft tissue image acquired at the previous time in the adjacent time and the pixel coordinates in the second deformation image belonging to the same orientation.

[0134] In some embodiments, the self-supervised training module 302 is further configured to: calculate a third loss value based on the similarity between the velocity vector of the three-dimensional point in the point cloud and the first deformation field; and / or calculate a fourth loss value based on the gradient change between the first deformation field and the second deformation field; and adjust the parameters of the neural network model based on at least one of the third loss value and the fourth loss value, the first loss value and the second loss value.

[0135] In some embodiments, the neural network model includes a first encoder, a second encoder, and a decoder; the first encoder is used to extract features from the point cloud contained in the sample to obtain point cloud features; the second encoder is used to extract features from the velocity vectors of three-dimensional points in the point cloud contained in the sample to obtain motion features; the decoder is used to perform deformation estimation processing based on the point cloud features and the motion features to obtain a velocity field.

[0136] In some embodiments, the neural network model is a U-shaped neural network model; wherein the first encoder and the second encoder are hop-connected, and the second encoder and the decoder are hop-connected.

[0137] In some embodiments, both the first encoder and the second encoder contain N neural network layers; N is an integer greater than 1; the neural network layers in the first encoder and the neural network layers in the second encoder are connected in a skip connection; the input data of the (n+1)th neural network layer in the first encoder is the output data of the nth neural network layer in the first encoder, and the input data of the (n+1)th neural network layer in the second encoder is the result of concatenating the output data of the nth neural network layer in the second encoder and the output data of the nth neural network layer in the first encoder, where 1 < n ≤ N.

[0138] In some embodiments, the decoder includes N neural network layers; N is an integer greater than 1; the input data of the first neural network layer of the decoder includes the output data of the last neural network layer in the second encoder; the input data of the (m+1)th neural network layer of the decoder is the result of concatenating the output data of the mth neural network layer in the decoder and the output data of the neural network layer corresponding to the (m+1)th neural network layer in the decoder, 1 < m ≤ N; the input data of the last neural network layer in the decoder also includes the velocity field estimated by the neural network model at the previous time step.

[0139] In some embodiments, the point cloud contained in any sample is obtained by: performing depth estimation on two soft tissue images acquired from two different orientations at the previous time in the adjacent time to obtain first depth information; using the first depth information and the intrinsic and extrinsic parameters of the image acquisition device, mapping the pixels in the soft tissue image acquired from one orientation at the previous time in the adjacent time from two-dimensional space to three-dimensional space to obtain the point cloud of the previous time in the adjacent time.

[0140] In some embodiments, the velocity vector of a 3D point in the point cloud contained in any sample is obtained by: performing depth estimation on two soft tissue images acquired from two different orientations at the previous time in the adjacent time to obtain first depth information; and performing depth estimation on two soft tissue images acquired from two different orientations at the next time in the adjacent time to obtain second depth information; performing optical flow estimation on two soft tissue images acquired from the same orientation at the adjacent time using an optical flow estimation network to obtain the velocity vector of the pixel; and projecting the velocity vector of the pixel using the first depth information and the second depth information to obtain the velocity vector of the 3D point in the point cloud.

[0141] In some embodiments, the soft tissue image is an image obtained by segmenting surgical instruments in a soft tissue image using a pre-trained instrument segmentation network.

[0142] In some embodiments, the four soft tissue images include left and right views acquired at the previous time step in adjacent time steps, and left and right views acquired at the next time step in adjacent time steps.

[0143] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0144] Corresponding to the aforementioned embodiments of the soft tissue deformation estimation method, this application also provides embodiments of a soft tissue deformation estimation device. The embodiments of the soft tissue deformation estimation device of this application can be applied to electronic devices, the hardware structure of which is similar to... Figure 7 Similarly, this will not be elaborated upon here.

[0145] Please see Figure 9 This application provides a soft tissue deformation estimation device, comprising:

[0146] The image acquisition module 401 is used to acquire four soft tissue images taken from two different orientations at adjacent times.

[0147] Image processing module 402 is used to perform depth estimation and optical flow estimation on the four soft tissue images to obtain the point cloud of the previous time step and the velocity vector of the three-dimensional points in the point cloud in adjacent time steps.

[0148] The deformation estimation module 403 is used to process the velocity vectors of the point cloud and the three-dimensional points in the point cloud into the soft tissue deformation estimation model to obtain the velocity field; wherein, the soft tissue deformation estimation model is trained based on the above method.

[0149] The soft tissue deformation information acquisition module 404 is used to integrate the velocity field to obtain soft tissue deformation information based on the integration result.

[0150] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0151] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0152] In some embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any of the above embodiments.

[0153] In some embodiments, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the method as described in any of the preceding embodiments.

[0154] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0155] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0156] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0157] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0158] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0159] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0160] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0161] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A training method for a soft tissue deformation estimation model, characterized in that, include: A training sample set is obtained, wherein different samples in the training sample set include point clouds at different times and velocity vectors of three-dimensional points in the point clouds; wherein any sample is obtained by depth estimation and optical flow estimation based on four soft tissue images acquired from two different orientations at adjacent times. The preset neural network model is trained in a self-supervised manner using samples from the training sample set to obtain a soft tissue deformation estimation model. The self-supervised training instruction repeats the following iterative process until the iteration termination condition is met: In each iteration, the velocity field output by the neural network model is obtained based on the samples in the training sample set, and the velocity field is integrated to obtain the first deformation field and the second deformation field that satisfy the differential homeomorphism mapping relationship. Two soft tissue images acquired at the previous moment in the adjacent time interval are deformed using the first deformation field to obtain two first deformed images. Then, the two first deformed images are deformed using the second deformation field to obtain two second deformed images. If the iteration termination condition is not met, the parameters of the neural network model are adjusted based on the four soft tissue images, the two first deformation images, and the two second deformation images to obtain the neural network model for the next iteration. The iteration termination conditions include: reaching a preset number of iterations, the difference between the first deformed image and the soft tissue image acquired at the next adjacent moment being less than a first preset difference, and / or the difference between the second deformed image and the soft tissue image acquired at the previous adjacent moment being less than a second preset difference. The parameters of the neural network model are adjusted according to the first loss value and the second loss value; The first loss value includes the photometric loss value between the pixel value of the soft tissue image acquired at the next time in the adjacent time and the pixel value of the first deformation image belonging to the same orientation. The second loss value includes the L1 norm distance between the pixel coordinates in the soft tissue image acquired at the previous time step in the adjacent time step and the pixel coordinates in the second deformation image belonging to the same orientation.

2. The method according to claim 1, characterized in that, The first deformation field characterizes the deformation from the previous moment to the next moment in the adjacent time intervals, and the second deformation field characterizes the deformation from the next moment to the previous moment in the adjacent time intervals.

3. The method according to claim 1 or 2, characterized in that, The learning objectives of the self-supervised training include: the first deformed image is the same as or similar to the soft tissue image acquired at the next adjacent time step, and the second deformed image is the same as or similar to the soft tissue image acquired at the previous adjacent time step.

4. The method according to claim 1, characterized in that, The step of using the first deformation field to deform two soft tissue images acquired at the previous time point in the adjacent time interval to obtain two first deformation images includes: For each soft tissue image acquired at the previous time in the adjacent time, the point cloud corresponding to the soft tissue image is deformed using the first deformation field to obtain the first deformed point cloud. Using the intrinsic and extrinsic parameters of the image acquisition device, the first deformation point cloud is projected onto a two-dimensional space to obtain the first deformation image corresponding to the soft tissue image; The step of using the second deformation field to deform two first deformed images to obtain two second deformed images includes: For each first deformed image, the second deformation field is used to deform the first deformed point cloud corresponding to the first deformed image to obtain the second deformed point cloud. Using the intrinsic and extrinsic parameters of the image acquisition device, the second deformation point cloud is projected into a two-dimensional space to obtain the second deformation image.

5. The method according to claim 1, characterized in that, The method further includes: A third loss value is calculated based on the similarity between the velocity vector of the three-dimensional points in the point cloud and the first deformation field; and / or, a fourth loss value is calculated based on the gradient change between the first deformation field and the second deformation field. The step of adjusting the parameters of the neural network model based on the first loss value and the second loss value further includes: The parameters of the neural network model are adjusted based on at least one of the third and fourth loss values, the first loss value, and the second loss value.

6. The method according to claim 1, characterized in that, The neural network model includes a first encoder, a second encoder, and a decoder; The first encoder is used to extract features from the point cloud contained in the sample to obtain point cloud features; The second encoder is used to extract features from the velocity vectors of three-dimensional points in the point cloud contained in the sample, and obtain motion features; The decoder is used to perform deformation estimation processing based on the point cloud features and the motion features to obtain the velocity field.

7. The method according to claim 6, characterized in that, The neural network model includes a U-shaped neural network model; wherein the first encoder and the second encoder are hop-connected, and the second encoder and the decoder are hop-connected.

8. The method according to claim 6 or 7, characterized in that, Both the first encoder and the second encoder contain N neural network layers; N is an integer greater than 1. The neural network layers in the first encoder and the neural network layers in the second encoder are skip connections; The input data of the (n+1)th neural network layer in the first encoder is the output data of the nth neural network layer in the first encoder, and the input data of the (n+1)th neural network layer in the second encoder is the result of concatenating the output data of the nth neural network layer in the second encoder and the output data of the nth neural network layer in the first encoder, where 1 < n ≤ N.

9. The method according to claim 8, characterized in that, The decoder comprises N neural network layers; N is an integer greater than 1. The neural network layer in the decoder and the neural network layer in the second encoder are connected in a skip connection; The input data of the first neural network layer of the decoder includes the output data of the last neural network layer in the second encoder; The input data of the (m+1)th neural network layer of the decoder is the result of concatenating the output data of the mth neural network layer in the decoder and the output data of the neural network layer in the decoder corresponding to the (m+1)th neural network layer in the decoder, where 1 < m ≤ N; The input data for the last neural network layer in the decoder also includes the velocity field estimated by the neural network model at the previous time step.

10. The method according to claim 1, characterized in that, The point cloud contained in any sample is obtained in the following way: Depth estimation is performed on two soft tissue images acquired from two different orientations at the previous time point in the adjacent time interval to obtain the first depth information; Using the first depth information and the internal and external parameters of the image acquisition device, the pixels in the soft tissue image acquired from one of the directions at the previous moment in the adjacent time are mapped from two-dimensional space to three-dimensional space to obtain the point cloud of the previous moment in the adjacent time.

11. The method according to claim 10, characterized in that, The velocity vectors of the 3D points in the point cloud contained in any sample are obtained in the following way: Depth estimation is performed on two soft tissue images acquired from two different orientations at the previous time point in the adjacent time period to obtain first depth information; and depth estimation is performed on two soft tissue images acquired from two different orientations at the next time point in the adjacent time period to obtain second depth information. Optical flow estimation is performed on two soft tissue images acquired from the same location at adjacent time points using an optical flow estimation network to obtain the velocity vector of the pixels; The velocity vectors of the pixels are projected and mapped using the first depth information and the second depth information to obtain the velocity vectors of the three-dimensional points in the point cloud.

12. The method according to claim 1, characterized in that, The soft tissue image is obtained by segmenting surgical instruments in the soft tissue image using a pre-trained instrument segmentation network.

13. The method according to claim 1, characterized in that, The four soft tissue images include left and right views acquired at the previous time step in adjacent time steps, and left and right views acquired at the next time step in adjacent time steps.

14. A method for estimating soft tissue deformation, characterized in that, include: Acquire four soft tissue images taken from two different locations at adjacent time points; Depth estimation and optical flow estimation are performed on the four soft tissue images to obtain the point cloud and the velocity vector of the three-dimensional points in the point cloud at the previous time in adjacent time intervals. The point cloud and the velocity vectors of the three-dimensional points in the point cloud are input into the soft tissue deformation estimation model for processing to obtain the velocity field; wherein, the soft tissue deformation estimation model is trained based on the method described in any one of claims 1 to 13; The velocity field is integrated to obtain soft tissue deformation information based on the integration result.

15. A training device for a soft tissue deformation estimation model, characterized in that, include: The training sample acquisition module is used to acquire a training sample set, wherein different samples in the training sample set include point clouds at different times and velocity vectors of three-dimensional points in the point clouds; wherein any sample is obtained by depth estimation and optical flow estimation based on four soft tissue images acquired from two different orientations at adjacent times. The self-supervised training module is used to perform self-supervised training on a preset neural network model using samples from the training sample set to obtain a soft tissue deformation estimation model. Specifically, the self-supervised training module is used to repeat the following iterative process until the iteration termination condition is met: In each iteration, the velocity field output by the neural network model is obtained based on the samples in the training sample set; the velocity field is integrated to obtain a first deformation field and a second deformation field that satisfy the differential homeomorphism mapping relationship; the first deformation field is used to deform two soft tissue images acquired at the previous time step in the adjacent time steps to obtain two first deformation images; the second deformation field is then used to deform the two first deformation images to obtain two second deformation images; if the iteration termination condition is not met, the parameters of the neural network model are adjusted according to the four soft tissue images, the two first deformation images, and the two second deformation images to obtain the neural network model in the next iteration. The iteration termination conditions include: reaching a preset number of iterations, the difference between the first deformed image and the soft tissue image acquired at the next adjacent moment being less than a first preset difference, and / or the difference between the second deformed image and the soft tissue image acquired at the previous adjacent moment being less than a second preset difference. The parameters of the neural network model are adjusted according to the first loss value and the second loss value; The first loss value includes the photometric loss value between the pixel value of the soft tissue image acquired at the next time in the adjacent time and the pixel value of the first deformation image belonging to the same orientation. The second loss value includes the L1 norm distance between the pixel coordinates in the soft tissue image acquired at the previous time step in the adjacent time step and the pixel coordinates in the second deformation image belonging to the same orientation.

16. A soft tissue deformation estimation device, characterized in that, include: The image acquisition module is used to acquire four soft tissue images taken from two different locations at adjacent time points; The image processing module is used to perform depth estimation and optical flow estimation on the four soft tissue images to obtain the point cloud of the previous time step and the velocity vector of the three-dimensional points in the point cloud in the adjacent time steps. The deformation estimation module is used to process the velocity vectors of the point cloud and the three-dimensional points in the point cloud into the soft tissue deformation estimation model to obtain the velocity field; wherein, the soft tissue deformation estimation model is trained based on the method described in any one of claims 1 to 13; The soft tissue deformation information acquisition module is used to integrate the velocity field to obtain soft tissue deformation information based on the integration result.

17. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method according to any one of claims 1 to 14.

18. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 14.

Images