Medical image processing method, system, device and storage medium

Abstract

The embodiments of the present application provide a medical image processing method, system, device and storage medium, which input a video image of a target tissue during surgery through an input module; a processing module determines a correspondence between the second video frame and the three-dimensional model based on a first video frame, a three-dimensional model of the target tissue and a second video frame, and aligns the three-dimensional model with the second video frame based on the correspondence, wherein the first video frame and the second video frame are respectively consecutive video frames; a display module displays the aligned second video frame and the three-dimensional model in real time, and can dynamically align the video frame and the three-dimensional model in real time based on the video image.

Description

Technical Field

[0001] The present application belongs to the field of medical image processing technology, and in particular, relates to a medical image processing method, system, device and storage medium. Background Art

[0002] Minimally invasive surgery has the advantages of less trauma, faster recovery, and reduced pain for patients. The endoscope, as the doctor's "eyes", can effectively help doctors see the lesions clearly during minimally invasive surgery. The surgical field of view under the endoscope is limited, and many lesion blood vessels are hidden under the surface of the organ, which places high demands on the doctor's ability and experience. Intraoperative navigation technology can map the preoperative model of the target organ during surgery to provide real-time guidance to the doctor. However, some navigation systems currently use additional hardware devices to achieve intraoperative navigation. The hardware devices may include: electromagnetic sensors or IMU sensors, but such sensors can usually only reflect the movement of the endoscope. When the endoscope moves, the preoperative model is rigidly registered with the intraoperative image according to the movement of the endoscope. However, when the endoscope is fixed and the organ moves, the existing intraoperative navigation technology cannot register the preoperative model with the intraoperative image. Summary of the invention

[0003] Based on this, the embodiments of the present application provide a medical image processing method, system, device and storage medium, which can perform real-time dynamic registration of video frames and three-dimensional models based on video images.

[0004] The present application embodiment provides a medical image processing system, including:

[0005] An input module, used for inputting a video image of a target tissue during surgery;

[0006] a processing module, configured to determine a correspondence between the second video frame and the three-dimensional model according to the first video frame, the three-dimensional model of the target tissue, and the second video frame, and to align the three-dimensional model with the second video frame according to the correspondence, wherein the first video frame and the second video frame are respectively consecutive video frames;

[0007] The display module is used to display the registered second video frame and the three-dimensional model in real time.

[0008] In some embodiments, the processing module is used to determine the correspondence between the second video frame and the three-dimensional model according to the first video frame, the three-dimensional model of the target tissue, and the second video frame, including:

[0009] Determine a vector field of pixel movement in the second video frame according to the first video frame and the second video frame;

[0010] determining an updated position of a control point in a second video frame according to the vector field;

[0011] The correspondence relationship between the second video frame and the three-dimensional model is determined according to the updated position of the control point and the correspondence relationship between the first video frame and the three-dimensional model.

[0012] In some embodiments, the processing module is further configured to:

[0013] The weight value of each feature point relative to the control point is updated according to the updated position of the control point in the second video frame.

[0014] In some embodiments, the processing module determines the updated position of the control point in the second video frame according to the vector field, including:

[0015] Mapping feature points in a feature point-dense area in the first video frame to the second video frame according to the vector field, and determining update positions of control points in the second video frame according to weight values ​​of the feature points relative to the control points, wherein the feature point-dense area is an area in the second video frame where the number of feature points is greater than a quantity threshold;

[0016] According to the vector field, control points in a feature point sparse region in the first video frame are mapped to the second video frame to obtain updated positions of control points in the feature point sparse region, where the number of feature points in the second video frame is less than the number threshold.

[0017] In some embodiments, the processing module determines the vector field of pixel movement in the second video frame according to the first video frame and the second video frame, including:

[0018] For any target pixel in the first video frame, searching for the target pixel in the second video frame;

[0019] According to the position of the target pixel in the first video frame and the position of the target pixel in the second video frame, a displacement field of the target pixel movement is determined to obtain a vector field of pixel movement in the second video frame.

[0020] In some embodiments, when the first video frame is a first frame, the processing module establishes a control point pair according to the registration result, including:

[0021] Uniformly sampling feature points from the first video frame to obtain initial control points;

[0022] Filtering target control points from the initial control points according to a registration result obtained after registering the three-dimensional model and the first video frame, wherein the target control points have corresponding control points in the three-dimensional model, so as to obtain the control point pair.

[0023] In some embodiments, the processing module is used to determine the correspondence between the second video frame and the three-dimensional model according to the first video frame, the three-dimensional model of the target tissue, and the second video frame, including:

[0024] Extracting feature points from the second video frame, and extracting feature points from the first video frame;

[0025] Matching the feature points extracted from the first video frame with the feature points extracted from the second video frame to obtain a matching result;

[0026] Determine an update position of the control point according to the matching result;

[0027] The correspondence relationship between the second video frame and the three-dimensional model is determined according to the update position, the correspondence relationship between the first video frame and the three-dimensional model.

[0028] In some embodiments, the processing module registers the three-dimensional model with the second video frame according to the corresponding relationship, including:

[0029] Determining the relative position relationship between the target tissue and the intraoperative image sensor and the elastic deformation of each feature point of the three-dimensional model according to the corresponding relationship between the second video frame and the three-dimensional model, wherein the intraoperative image sensor is used to collect video images of the target tissue during surgery;

[0030] The three-dimensional model and the second video frame are registered according to the elastic deformation and the relative position relationship.

[0031] In some embodiments, determining the relative position relationship between the target tissue and the intraoperative image sensor and the elastic deformation of each feature point of the three-dimensional model according to the corresponding relationship between the second video frame and the three-dimensional model includes:

[0032] Determining a relative position relationship between the target tissue and the intraoperative image sensor according to the corresponding relationship;

[0033] Determining displacement change information of rigid motion of the three-dimensional model according to the relative position relationship;

[0034] Determining a virtual force causing a change in the three-dimensional model according to the displacement change information;

[0035] Simulating the elastic deformation of each feature point in the three-dimensional model according to the virtual force and the constraint conditions of the elastic deformation to obtain the elastic deformation of each feature point in the three-dimensional model, or;

[0036] The elastic deformation of each characteristic point in the three-dimensional model is calculated using a finite element analysis method according to the virtual force and the biomechanical model of the target tissue.

[0037] The present application provides a medical image processing method, including:

[0038] Acquire video images of target tissue during surgery;

[0039] Determine a correspondence between the second video frame and the three-dimensional model according to the first video frame, the three-dimensional model of the target tissue, and the second video frame, and align the three-dimensional model with the second video frame according to the correspondence, wherein the first video frame and the second video frame are respectively consecutive video frames;

[0040] The registered second video frame and the three-dimensional model are displayed in real time.

[0041] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described methods when executing the computer program.

[0042] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned methods is implemented.

[0043] An embodiment of the present application provides a computer program product. When the computer program product is executed on a terminal device, the electronic device executes any one of the above methods.

[0044] A medical image processing system provided by an embodiment of the present application inputs a video image of a target tissue during surgery through an input module; a processing module determines a correspondence between the second video frame and the three-dimensional model based on a first video frame, a three-dimensional model of the target tissue, and a second video frame, and aligns the three-dimensional model with the second video frame based on the correspondence, wherein the first video frame and the second video frame are respectively consecutive video frames; a display module displays the aligned second video frame and the three-dimensional model in real time, and can dynamically align the video frame and the three-dimensional model in real time based on the video image. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Hereinafter, the present application will be described in more detail according to embodiments and with reference to the accompanying drawings.

[0046] Figure 1 A schematic diagram of the structure of a medical image processing system provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of a medical image processing method for implementing this application;

[0048] Figure 3 A schematic diagram of a process flow of a medical image processing method provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0050] In the drawings, the same reference numerals are used for the same components, and the drawings are not drawn to scale. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0052] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0053] If similar descriptions of "first\second\third" appear in the application documents, the following instructions are added. In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0055] Based on the problems in the related art, an embodiment of the present application provides a medical image processing system. An embodiment of the present application provides a medical image processing system. The modules included in the system and the units included in each module can be implemented by a processor in a computer device; of course, they can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU, Central Processing Unit), a microprocessor (MPU, Micro Processor Unit), a digital signal processor (DSP, Digital Signal Processing) or a field programmable gate array (FPGA, Field Programmable Gate Array), etc.

[0056] The present application embodiment provides a medical image processing system. Figure 1 A schematic diagram of the structure of a medical image processing system provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the medical image processing system 100 includes: an input module 101, a processing module 102 and a display module 103, the input module is used to input a video image of a target tissue during surgery; the processing module is used to determine a correspondence between the second video frame and the three-dimensional model according to a first video frame, the three-dimensional model of the target tissue and a second video frame, and to align the three-dimensional model with the second video frame according to the correspondence, the first video frame and the second video frame being consecutive video frames respectively; the display module is used to display the aligned second video frame and the three-dimensional model in real time.

[0057] In the embodiment of the present application, the target organ may be an organ in the human body, and the target organ may be: liver, heart, stomach, etc.

[0058] In the embodiment of the present application, during the operation, the surgical operator can use the image acquisition module to collect the intraoperative image of the target organ in real time, thereby obtaining a video image. The image acquisition module can be an endoscope. The video image includes two consecutive frames of images, and any two consecutive frames can be the first video frame and the second video frame.

[0059] In the embodiment of the present application, the three-dimensional model of the target tissue may be a three-dimensional model of the target tissue established before surgery.

[0060] In the embodiment of the present application, the processing module can determine the correspondence between the second video frame and the three-dimensional model according to the pixel movement vector field of the video frame, and the pixel movement vector field can be determined based on the first video frame and the second video frame.

[0061] In some embodiments, the processing module may also use feature matching to determine the correspondence between the second video frame and the three-dimensional model.

[0062] In an embodiment of the present application, the correspondence between the second video frame and the three-dimensional model can be a correspondence between features in the second video frame and features in the three-dimensional model, and since the features in the second video frame correspond to pixel points, the correspondence can also be a correspondence between pixel points in the second video frame and features in the three-dimensional model.

[0063] In the embodiment of the present application, the registration may include: rigid registration and elastic registration.

[0064] In the embodiment of the present application, rigid registration is a method of matching two data sets so that they are aligned in the same coordinate system. When the second video frame and the three-dimensional model are rigidly registered, the three-dimensional model can be rotated, translated, scaled, and other rigid transformations to make the three-dimensional model and the second video frame match each other as much as possible in spatial position. This registration is suitable for situations where the position and shape must remain unchanged, and can help doctors align previously planned surgical plans with the actual patient's anatomical structure.

[0065] In the embodiment of the present application, elastic registration is used to more accurately and flexibly match information from different data sources, taking into account more complex deformations and deformations of the target organ. The three-dimensional model can be deformed nonlinearly and locally changed to achieve elastic registration of the three-dimensional model and the second video frame. In the embodiment of the present application, the three-dimensional model planned by the doctor before the operation is elastically registered with the intraoperative image in order to more finely align the structure between the two to provide more accurate guidance and auxiliary information.

[0066] In the embodiment of the present application, the display module can display the registered three-dimensional model and the second video frame in real time.

[0067] The display module displaying the registered three-dimensional model and the second video frame may include: displaying the three-dimensional model and the second video frame in a superimposed manner.

[0068] A medical image processing system provided by an embodiment of the present application inputs a video image of a target tissue during surgery through an input module; a processing module determines a correspondence between the second video frame and the three-dimensional model based on a first video frame, a three-dimensional model of the target tissue, and a second video frame, and aligns the three-dimensional model with the second video frame based on the correspondence, wherein the first video frame and the second video frame are respectively consecutive video frames; a display module displays the aligned second video frame and the three-dimensional model in real time, and can dynamically align the video frame and the three-dimensional model in real time based on the video image.

[0069] In the embodiment of the present application, since video images can be collected through an endoscope, the video frames and the three-dimensional model can be aligned through the video images, and the alignment can be achieved without the aid of other hardware devices (such as electromagnetic sensors or IMU sensors). In addition, when the laparoscope is fixed and the organ moves, the three-dimensional model and the video frame can also be aligned to achieve accurate tracking of the movement of the target organ itself.

[0070] In some embodiments, the processing module is used to determine the correspondence between the second video frame and the three-dimensional model according to the first video frame, the three-dimensional model of the target tissue, and the second video frame, including:

[0071] The processing module determines a vector field of pixel movement in the second video frame according to the first video frame and the second video frame, determines an updated position of a control point in the second video frame according to the vector field, and determines a corresponding relationship between the second video frame and the three-dimensional model according to the updated position of the control point and the corresponding relationship between the first video frame and the three-dimensional model.

[0072] In the embodiment of the present application, the vector field of pixel movement is used to describe the movement of pixels between two video frames. The displacement of the pixels can be calculated by comparing the position changes of the pixels in the two video frames, and the vector field of movement can be determined by the displacement.

[0073] In the embodiment of the present application, the control point may be a feature point set in the video frame.

[0074] In an embodiment of the present application, the correspondence between the first video frame and the three-dimensional model may be established based on a video frame before the first video frame. In some embodiments, if the first video frame is the first frame of a video image, the correspondence between the first video frame and the three-dimensional model may be obtained after preliminary registration of the three-dimensional model and the first video frame.

[0075] In the embodiment of the present application, the approximate correspondence between the second video frame and the three-dimensional model can be inferred based on the updated position of the control point combined with the correspondence between the first video frame and the three-dimensional model.

[0076] In some embodiments, the processing module determines the vector field of pixel movement in the second video frame based on the first video frame and the second video frame, which may include:

[0077] The processing module searches for any target pixel in the first video frame on the second video frame, and determines a displacement field of movement of the target pixel according to a position of the target pixel in the first video frame and a position of the target pixel in the second video frame, so as to obtain a vector field of pixel movement in the second video frame.

[0078] In the embodiment of the present application, the target pixel may be any pixel. In the embodiment of the present application, when searching for the target pixel on the second video frame, the search and matching may be performed by a sliding window, template matching, feature matching, or the like.

[0079] Exemplarily, taking feature matching as an example, feature extraction can be performed on the target pixel in the first video frame, and features such as color, texture, shape, etc. can be selected. Using the selected features, search for pixels similar to the target pixel in the first video frame in the second video frame. A similarity measurement method (such as Euclidean distance, cosine similarity, etc.) can be used to compare the similarity between the target pixel and the candidate pixel in the second video frame. In the second video frame, the pixel closest to the target pixel in the first video frame is found according to the similarity measurement, so as to search for the target pixel on the second video frame.

[0080] In the embodiment of the present application, the processing module can find the position of the target pixel in the second video frame by searching for the target pixel in the second video frame. By comparing the position difference of the target pixel in the two video frames, the displacement of the target pixel in space can be calculated to form a displacement field. By combining the displacement fields of all target pixels together, a vector field of pixel movement in the second video frame can be obtained, and the movement of pixels in the entire video frame can be described by the vector field.

[0081] In an embodiment of the present application, the displacement fields of all target pixels are combined together to obtain the vector of pixel movement in the second video frame, which can be achieved in the following manner: for each target pixel, the corresponding displacement vector is calculated according to the position difference between the first video frame and the second video frame. The pixels in the video frame are divided into grids or grid points so that the displacement information can be applied to the entire picture. Based on the known displacement information of the target pixels, the displacement information of the remaining pixels on the grid is estimated by an interpolation method (such as bilinear interpolation). The displacement information of each pixel point is combined into a vector field of overall pixel movement, thereby describing the movement of the pixels in the entire second video frame.

[0082] In the embodiment of the present application, through such a method, the motion information of a single pixel can be integrated into the entire image, so as to have a more comprehensive understanding of the movement of pixels in the video frame.

[0083] In some embodiments, the processing module determining the updated position of the control point in the second video frame according to the vector field may include:

[0084] The processing module maps the feature points in the feature point dense area in the first video frame to the second video frame according to the vector field, and determines the update position of the control point in the second video frame according to the weight value of the feature point relative to the control point. The feature point dense area is an area in the second video frame where the number of feature points is greater than a quantity threshold.

[0085] In the embodiment of the present application, the weight value of the feature point relative to the control point is determined based on the first video frame, and the feature point can be searched within the field of each control point in the first video frame with a preset radius, and the weight of the feature point corresponding to each control point is determined based on the distance between the feature point searched within the field of each control point and the corresponding control point. In the embodiment of the present application, the closer the distance, the greater the weight, and the farther the distance, the smaller the weight, and the preset radius can be configured.

[0086] In an embodiment of the present application, mapping feature points in a feature point dense area in the first video frame to the second video frame according to the vector field includes: selecting some key feature points in the dense feature point area in the first video frame according to the calculated vector field of pixel movement, and determining the positions of these feature points in the second video frame by mapping each feature point to the second video frame according to the movement in the vector field.

[0087] In the embodiment of the present application, for each feature point, its weight value at the position can be determined according to its distance relative to the surrounding control points or other features. Generally, the closer the control point is, the greater the influence on the feature point is. The updated position of the control point in the second video frame is determined according to the position of the feature point in the second video frame and its weight value relative to the control point. The weighted feature point information can be used to effectively determine the position of the control point to better map and track the entire area.

[0088] In the embodiment of the present application, the position of the control point is determined in a dense area by using feature points in the area, so that the accuracy of the determined position of the control point can be higher.

[0089] The processing module maps the control points in the feature point sparse area in the first video frame to the second video frame according to the vector field to obtain the updated positions of the control points in the feature point sparse area, where the number of feature points in the second video frame is less than the number threshold.

[0090] In the embodiment of the present application, the vector field of pixel movement can be used to map the control points in the sparse feature point area in the first video frame to the second video frame, so as to determine the positions of these control points in the second video frame.

[0091] In an embodiment of the present application, for sparse areas, the neighborhood of the control point may not be able to search for feature points, or the accuracy of the feature points cannot be guaranteed. Therefore, the control point and the feature point are decoupled, and the control point is mapped from the first image frame to the second image frame according to the vector field to directly obtain the updated position of the updated control point.

[0092] In the embodiment of the present application, the sparse area may be an area where the surface of the target organ is smooth or the anatomical structure is not visible under the laparoscope.

[0093] In some embodiments, after the processing module aligns the three-dimensional model with the second video frame according to the corresponding relationship, the processing module is further used to update the weight value of each feature point relative to the control point according to the updated position of the control point in the second video frame.

[0094] In an embodiment of the present application, feature points can be searched within the area of ​​each control point in the second video frame with a preset radius, and the weights of the feature points corresponding to each control point can be determined based on the distances between the feature points searched within the area of ​​each control point and the corresponding control points, so that the weight values ​​of each feature point relative to the control points can be updated.

[0095] In some embodiments, when the first video frame is the first frame, the processing module establishes a control point pair according to the registration result, including:

[0096] The processing module uniformly samples feature points from the first video frame to obtain initial control points, and filters out target control points from the initial control points according to a registration result obtained after aligning the three-dimensional model with the first video frame to obtain the control point pair, wherein the target control point has a corresponding control point in the three-dimensional model.

[0097] In the embodiment of the present application, uniform sampling of feature points is performed in the first video frame, and some key feature points can be identified by various feature detection algorithms (such as SIFT, SURF, etc.) to obtain initial control points.

[0098] In the embodiment of the present application, the three-dimensional model and the first video frame may be registered, and the registration may include: rigid registration and / or elastic registration. After the registration, there is a corresponding relationship between the feature points in each first video frame and the feature points of the three-dimensional model.

[0099] In the embodiment of the present application, the target control point corresponding to the initial control point can be found in the registered three-dimensional model, thereby obtaining a control point pair.

[0100] In the embodiment of the present application, after the control point pair is obtained, the control point pair can be used for subsequent updated position tracking and deformation estimation.

[0101] In the related art, when performing position tracking, one can only rely on the anatomical features or image features of the target organ itself, which has a narrow scope of application and is prone to failure. It is difficult to detect feature points on the smooth surface of some target organs, and the uneven distribution of such features will affect the accuracy of intraoperative navigation. In the embodiment of the present application, by setting control points, accurate registration can still be achieved when the feature points fail.

[0102] In the related art, in order to perform alignment, tracking markers are installed on the surface of the organ to achieve intraoperative tracking of the target organ. However, the method in the related art introduces additional surgical procedures and is only applicable to the tracking of rigid organs such as bones. It is impossible to fix markers on soft tissue organs such as the liver. The method provided in the embodiment of the present application performs alignment by setting control points, does not rely on additional visual markers, and can achieve accurate intraoperative tracking of both rigid organs and soft tissue organs.

[0103] In some embodiments, a processing module is used to determine the correspondence between the second video frame and the three-dimensional model based on the first video frame, the three-dimensional model of the target tissue and the second video frame, and may include: the processing module extracts feature points from the second video frame and extracts feature points from the first video frame, matches the feature points extracted from the first video frame with the feature points extracted from the second video frame to obtain a matching result, determines an updated position of the control point based on the matching result, and determines the correspondence between the second video frame and the three-dimensional model based on the updated position and the correspondence between the first video frame and the three-dimensional model.

[0104] In the embodiment of the present application, feature points can be extracted from the first video frame and the second video frame, and the feature point extraction in the first video frame and the second video frame can be achieved by various feature extraction algorithms such as SIFT, SURF, ORB, etc.

[0105] In the embodiment of the present application, the similarity between the feature points extracted from the first video frame and the feature points extracted from the second video frame can be calculated for matching. The greater the similarity, the closer the match, and the smaller the similarity, the less the match.

[0106] In the embodiment of the present application, since the feature points include control points, the update positions of the control points can be determined from the matched feature point position information.

[0107] In the embodiment of the present application, the approximate correspondence between the second video frame and the three-dimensional model can be inferred based on the updated position of the control point combined with the correspondence between the first video frame and the three-dimensional model.

[0108] In some embodiments, the processing module registers the three-dimensional model with the second video frame according to the corresponding relationship, including:

[0109] The processing module determines the relative position relationship between the target tissue and the intraoperative image sensor, and the elastic deformation of each feature point of the three-dimensional model according to the corresponding relationship between the second video frame and the three-dimensional model, wherein the intraoperative image sensor is used to capture video images of the target tissue during surgery, and the three-dimensional model and the second video frame are aligned according to the elastic deformation and the relative position relationship.

[0110] In the embodiment of the present application, since the second video frame is acquired by the intraoperative image sensor, the relative position relationship between the target tissue and the intraoperative image sensor can be determined through the relationship between the second video frame and the three-dimensional model.

[0111] In some embodiments, the processing module determines the relative position relationship between the target tissue and the intraoperative image sensor and the elastic deformation of each feature point of the three-dimensional model according to the corresponding relationship between the second video frame and the three-dimensional model, wherein the intraoperative image sensor is used to collect the video image of the target tissue during the operation, which may include:

[0112] The processing module determines the relative position relationship between the target tissue and the intraoperative image sensor according to the corresponding relationship, determines the displacement change information of the rigid movement of the three-dimensional model according to the relative position relationship, determines the virtual force caused by the change of the three-dimensional model according to the displacement change information, and simulates the elastic deformation of each feature point in the three-dimensional model according to the virtual force and the constraint conditions of the elastic deformation to obtain the elastic deformation of each feature point in the three-dimensional model, or; calculates the elastic deformation of each feature point in the three-dimensional model by using a finite element analysis method according to the virtual force and the biomechanical model of the target tissue.

[0113] In the embodiment of the present application, since the second video frame is acquired by the intraoperative sensor, the relative position transformation can be solved based on the correspondence and geometric transformation theory to determine the relative position relationship between the target tissue and the intraoperative image sensor. The geometric transformation theory can include homography matrix, essential matrix, etc.

[0114] In the embodiment of the present application, the displacement change information may include: changes such as translation and rotation of the entire model.

[0115] In an embodiment of the present application, a virtual force field matching the target tissue can be designed and created based on the geometric shape and material properties of the target tissue. The virtual force field can include various types of forces, such as constraint force, spring force, friction force, etc. After the virtual force field is created, the virtual force that causes changes in the three-dimensional model can be simulated through the virtual force field through displacement change information.

[0116] In the embodiment of the present application, an appropriate elastic model can be determined according to the material properties of the target tissue and the position of the characteristic points. This can include a linear elastic model, a nonlinear elastic model, etc., and a suitable model is selected according to the characteristics of the target tissue. According to the constraints of the virtual force and the elastic deformation, the constraints of each characteristic point in the simulation process are set to simulate the external constraints and internal constraints of the target tissue. According to the virtual force and the constraints, simulation calculations can be performed to simulate the elastic deformation of each characteristic point in the three-dimensional model.

[0117] In the embodiment of the present application, the biomechanical model is used to describe the mechanical properties of the target tissue, and the biomechanical model may include: an elastic model and a constitutive model of the material. Then, the elastic deformation of each feature point in the three-dimensional model is calculated using a finite element analysis method based on the virtual force and the biomechanical model of the target tissue.

[0118] In the embodiment of the present application, the three-dimensional model and the second video frame can be rigidly registered according to the relative position key, and the three-dimensional model and the second video frame can be elastically registered according to the elastic deformation.

[0119] The method provided in the embodiment of the present application can achieve real-time registration of both rigid and elastic movements of the target organ.

[0120] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application.

[0121] Based on the aforementioned medical image processing system, the application embodiment provides a medical image processing method that can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), medical imaging devices, etc. The application embodiment does not impose any restrictions on the specific types of electronic devices.

[0122] The functions implemented by the medical image processing method provided in the embodiment of the present application can be implemented by calling program codes by a processor of an electronic device, wherein the program codes can be stored in a computer storage medium.

[0123] The present application embodiment provides a medical image processing method. Figure 2 A schematic diagram of the implementation flow of a medical image processing method provided for the implementation of this application is as follows: Figure 2 As shown, the medical image processing method includes:

[0124] Step S1, obtaining a video image of a target tissue during surgery;

[0125] Step S2, determining a correspondence between the second video frame and the three-dimensional model according to the first video frame, the three-dimensional model of the target tissue, and the second video frame, and registering the three-dimensional model with the second video frame according to the correspondence, wherein the first video frame and the second video frame are respectively consecutive video frames;

[0126] Step S3, displaying the registered second video frame and the three-dimensional model in real time.

[0127] A medical image processing method provided by an embodiment of the present application obtains a video image of a target tissue during surgery; determines a correspondence between the second video frame and the three-dimensional model based on a first video frame, a three-dimensional model of the target tissue, and a second video frame; and aligns the three-dimensional model with the second video frame based on the correspondence, wherein the first video frame and the second video frame are respectively consecutive video frames; and displays the aligned second video frame and the three-dimensional model in real time, so as to dynamically align the video frame and the three-dimensional model in real time based on the video image.

[0128] In some embodiments, determining the correspondence between the second video frame and the three-dimensional model according to the first video frame, the three-dimensional model of the target tissue, and the second video frame includes:

[0129] Determine a vector field of pixel movement in the second video frame according to the first video frame and the second video frame;

[0130] determining an updated position of a control point in a second video frame according to the vector field;

[0131] The correspondence relationship between the second video frame and the three-dimensional model is determined according to the updated position of the control point and the correspondence relationship between the first video frame and the three-dimensional model.

[0132] In some embodiments, the method further comprises:

[0133] The weight value of each feature point relative to the control point is updated according to the updated position of the control point in the second video frame.

[0134] In some embodiments, determining the updated position of the control point in the second video frame according to the vector field includes:

[0135] Mapping feature points in a feature point-dense area in the first video frame to the second video frame according to the vector field, and determining update positions of control points in the second video frame according to weight values ​​of the feature points relative to the control points, wherein the feature point-dense area is an area in the second video frame where the number of feature points is greater than a quantity threshold;

[0136] According to the vector field, control points in a feature point sparse region in the first video frame are mapped to the second video frame to obtain updated positions of control points in the feature point sparse region, where the number of feature points in the second video frame is less than the number threshold.

[0137] In some embodiments, determining the vector field of pixel movement in the second video frame according to the first video frame and the second video frame includes:

[0138] For any target pixel in the first video frame, searching for the target pixel in the second video frame;

[0139] According to the position of the target pixel in the first video frame and the position of the target pixel in the second video frame, a displacement field of the target pixel movement is determined to obtain a vector field of pixel movement in the second video frame.

[0140] In some embodiments, when the first video frame is a first frame, establishing a control point pair according to the registration result includes:

[0141] Uniformly sampling feature points from the first video frame to obtain initial control points;

[0142] A target control point is screened out from the initial control points according to a registration result obtained after registering the three-dimensional model with the first video frame to obtain the control point pair, wherein the target control point has a corresponding control point in the three-dimensional model.

[0143] In some embodiments, determining the correspondence between the second video frame and the three-dimensional model according to the first video frame, the three-dimensional model of the target tissue, and the second video frame includes:

[0144] Extracting feature points from the second video frame, and extracting feature points from the first video frame;

[0145] Matching the feature points extracted from the first video frame with the feature points extracted from the second video frame to obtain a matching result;

[0146] Determine an update position of the control point according to the matching result;

[0147] The correspondence relationship between the second video frame and the three-dimensional model is determined according to the update position, the correspondence relationship between the first video frame and the three-dimensional model.

[0148] In some embodiments, registering the three-dimensional model with the second video frame according to the corresponding relationship includes:

[0149] Determining the relative position relationship between the target tissue and the intraoperative image sensor and the elastic deformation of each feature point of the three-dimensional model according to the corresponding relationship between the second video frame and the three-dimensional model, wherein the intraoperative image sensor is used to collect video images of the target tissue during surgery;

[0150] The three-dimensional model and the second video frame are registered according to the elastic deformation and the relative position relationship.

[0151] In some embodiments, determining the relative position relationship between the target tissue and the intraoperative image sensor and the elastic deformation of each feature point of the three-dimensional model according to the corresponding relationship between the second video frame and the three-dimensional model includes:

[0152] Determining a relative position relationship between the target tissue and the intraoperative image sensor according to the corresponding relationship;

[0153] Determining displacement change information of rigid motion of the three-dimensional model according to the relative position relationship;

[0154] Determining a virtual force causing a change in the three-dimensional model according to the displacement change information;

[0155] Simulating the elastic deformation of each feature point in the three-dimensional model according to the virtual force and the constraint conditions of the elastic deformation to obtain the elastic deformation of each feature point in the three-dimensional model, or;

[0156] The elastic deformation of each characteristic point in the three-dimensional model is calculated using a finite element analysis method according to the virtual force and the biomechanical model of the target tissue.

[0157] The working process of the above method can refer to the corresponding process in the aforementioned system embodiment, which will not be repeated here.

[0158] Based on the medical image processing method provided in the above-mentioned embodiment, the embodiment of the present application further provides a medical image processing method. Figure 3 A schematic diagram of a medical image processing method provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, including:

[0159] Step S11, performing initial registration of the three-dimensional model and the intraoperative image frame.

[0160] In the embodiment of the present application, initial rigid registration can be achieved by finding the corresponding key features on the three-dimensional model and the intraoperative image frame. Different anatomical features are defined for different target organs, and the corresponding anatomical key points / key lines can be extracted on the three-dimensional model and the intraoperative image frame using a deep learning network, and the initial registration results are obtained using these corresponding key anatomical structure calculation points and their projected positions in the two-dimensional image. The registration results include: the corresponding relationship between the image frame and the three-dimensional model.

[0161] Step S12: initializing control points.

[0162] In an embodiment of the present application, uniform sampling points may be set at a fixed density on the intraoperative image frame, the uniform sampling points may be screened according to the corresponding relationship obtained by the initial registration, and the uniform sampling points in the target organ may be used as control points.

[0163] Extract general feature points from intraoperative image frames, including but not limited to anatomical feature points of target organs, image corner points, etc. Search in the neighborhood of the control point with a fixed radius, and couple the control point with adjacent feature points based on information such as distance, that is, calculate the weight of the feature point in the neighborhood relative to the control point.

[0164] Step S13, the position of the control point is updated.

[0165] In the embodiment of the present application, when motion occurs in the field of view, that is, when the first video frame changes to the second video frame, the position of the existing control point is updated.

[0166] In the embodiment of the present application, the laparoscopic surgery video generally has a frame rate of more than 30 Hz and a very fast update speed, so the pixel points of the image can be regarded as unchanged in color, brightness, etc. between two adjacent frames. When an organ moves within the field of view, it will cause the corresponding pixel points on the video frame to move, and the vector field of pixel movement can reflect the speed of organ movement. Therefore, the local search method can be used to calculate the vector field of pixel movement. For any target pixel on the first video frame, the pixel points on the second video frame are searched within a neighborhood of a preset size to obtain the corresponding relationship between the same pixel point on the previous and next video frames. The vector field of pixel movement can be obtained by completing the search of all pixel points. After determining the vector field of pixel movement, the update position of the control point can be determined.

[0167] In the embodiment of the present application, for a feature point dense area, the position of the control point is determined by the feature points in the neighborhood. The feature points can be mapped from the first image frame to the second image frame according to the vector field, and the updated position of the control point can be obtained by using the coupling relationship.

[0168] In an embodiment of the present application, for a sparse feature point area, the sparse feature point area may include: an area with a smooth surface of the target organ or an area where the anatomical structure is not visible under the laparoscope, and the tracking scheme based on anatomical features will fail. In an embodiment of the present application, for a sparse feature point area, the neighborhood of the control point may not be able to search for the feature point, or the accuracy of the feature point cannot be guaranteed, so the control point and the feature point are decoupled. The control point is mapped from the first image frame to the second image frame according to the vector field, and the updated control point position is directly obtained. The first image frame may be the nth image frame, and the second image frame may be the n+1 image frame.

[0169] In the embodiment of the present application, after determining the update position of the control point, the coupling relationship of the control point can be updated: re-extracting the general feature points on the second image frame, and recalculating the coupling relationship between the control point and the feature point.

[0170] Step S14, performing dynamic registration.

[0171] In the embodiment of the present application, after determining the update position of the control point, the feature points on the three-dimensional model are kept in the same correspondence with the feature points of the first video frame, and the updated position can be used to achieve dynamic alignment of the three-dimensional model and the second video frame.

[0172] In the embodiment of the present application, dynamic registration includes: rigid registration and elastic registration.

[0173] In an embodiment of the present application, after determining the updated position of the control point, a new three-dimensional model and corresponding points in the second video frame are obtained, and the updated point pairs are used to calculate the perspective-n-Point (PnP) to obtain the new relative position relationship between the target tissue and the laparoscope, driving the preoperative model to undergo rigid motion to achieve rigid alignment.

[0174] In an embodiment of the present application, after the control points are updated, the displacement field of all control points of the N+1th frame relative to the Nth frame can be obtained, and the elastic deformation of the model is simulated using the position dynamics method PBD, and distance constraints and volume constraints are imposed to drive the preoperative model to undergo elastic deformation, thereby achieving elastic alignment.

[0175] The method provided in the embodiment of the present application uses initial registration to obtain the initial correspondence between the target tissue on the preoperative model and the intraoperative image, and maps the control points on the intraoperative image to the preoperative model to drive the deformation of the three-dimensional model.

[0176] In the embodiment of the present application, control points are formed by uniformly sampling on the image frame. When the characteristic anatomical structure of the target organ is not visible or the feature fails, the control points are decoupled from the anatomical features and can independently drive the deformation of the 3D model. Moreover, the density of the sampling points can be flexibly adjusted for different accuracy requirements to achieve high-precision and high-efficiency dynamic tracking.

[0177] The method provided in the embodiment of the present application establishes the corresponding relationship between the previous and next video frames in pixels when the image changes, maps the preoperative and intraoperative corresponding relationship of the previous moment to the current moment, updates the control points, and drives the model to obtain a new registration result, which can realize both rigid registration and elastic registration. The method quickly obtains new corresponding relationships based on time series information, realizes real-time dynamic registration, and can track different forms of organ movement in real time.

[0178] In some embodiments, when setting control points, the control points are screened. By screening the control points, the movement of complex organ tissues around the target tissue can be avoided to affect the tracking of the target tissue. Therefore, it can also be achieved by segmenting the target tissue. The target tissue can be segmented by traditional methods or by using a deep learning network. The objects to be segmented include but are not limited to the target organ, surrounding tissue organs, instruments, etc.

[0179] In some embodiments, when updating the position of the control point, the calculation of the vector field is to obtain the corresponding feature points on the next frame, so the matching of feature points can also be achieved by extracting features from the previous and next frames and then comparing them. The matching of feature points can be done by traditional methods, by calculating feature descriptors and then comparing them, or by directly using deep learning networks such as Superglue to achieve accurate matching.

[0180] In some embodiments, during dynamic registration, position-based dynamics (PBD) can be used to perform physical simulation on the model. The advantages of physical simulation are fast calculation speed and good real-time performance. In order to improve the accuracy of elastic registration, biomechanical modeling of the target tissue can be performed, the material of the target organ can be simulated using nonlinear elastic models, porous elastic models, etc., and the elastic deformation of the organ driven by the control point can be calculated using finite elements.

[0181] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 4 As shown, the electronic device 3 of this embodiment may include: at least one processor 30 ( Figure 4 Only one processor 30 is shown in the figure), a memory 31, and a computer program 32 stored in the memory 31 and executable on at least one processor 30. When the processor 30 executes the computer program 32, the steps in any of the above-mentioned method embodiments are implemented, for example Figure 2 Steps S1 to S3 in the illustrated embodiment. Alternatively, when the processor 30 executes the computer program 32, the functions of each module / unit in the above-mentioned system embodiments are realized, for example Figure 1 The functions of modules 101 to 103 are shown.

[0182] Exemplarily, the computer program 32 may be divided into one or more modules / units, one or more modules / units are stored in the memory 31, and are executed by the processor 30 to complete the present application. One or more modules / units may be a series of computer program 32 instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 32 in the electronic device 3.

[0183] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program 32. When the computer program 32 is executed by the processor 30, the steps in the above-mentioned method embodiments can be implemented.

[0184] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0185] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. According to this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program 32, and the computer program 32 can be stored in a computer-readable storage medium. When the computer program 32 is executed by the processor 30, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program 32 includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include at least: any entity or device capable of carrying the computer program code to the terminal, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0186] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0187] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0188] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0189] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A medical image processing system, characterized in that: include: An input module, used for inputting a video image of a target tissue during surgery; A processing module, used for determining a correspondence between the second video frame and the three-dimensional model according to a first video frame, the three-dimensional model of the target tissue and a second video frame, and registering the three-dimensional model with the second video frame according to the correspondence, wherein the first video frame and the second video frame are respectively consecutive video frames, including: determining a vector field of pixel movement in the second video frame according to the first video frame and the second video frame; determining an updated position of a control point in the second video frame according to the vector field; determining a corresponding relationship between the second video frame and the three-dimensional model according to the updated position of the control point and the correspondence between the first video frame and the three-dimensional model; The determining the update position of the control point in the second video frame according to the vector field comprises: mapping the feature points in the feature point dense area in the first video frame to the second video frame according to the vector field, determining the update position of the control point in the second video frame according to the weight value of the feature point relative to the control point, the feature point dense area is a region in the second video frame where the number of feature points is greater than a quantity threshold, mapping the control points in the feature point sparse area in the first video frame to the second video frame according to the vector field, and obtaining the update position of the control point in the feature point sparse area, the feature point sparse area is a region in the second video frame where the number of feature points is less than the quantity threshold; The display module is used to display the registered second video frame and the three-dimensional model in real time.

2. The medical image processing system according to claim 1, characterized in that: The processing module is also used for: The weight value of each feature point relative to the control point is updated according to the updated position of the control point in the second video frame.

3. The medical image processing system according to claim 1, characterized in that: The processing module determines a vector field of pixel movement in the second video frame according to the first video frame and the second video frame, including: For any target pixel in the first video frame, searching for the target pixel in the second video frame; According to the position of the target pixel in the first video frame and the position of the target pixel in the second video frame, a displacement field of the target pixel movement is determined to obtain a vector field of pixel movement in the second video frame.

4. The medical image processing system according to claim 1, characterized in that: When the first video frame is a first frame, the processing module establishes a control point pair according to the registration result, including: Uniformly sampling feature points from the first video frame to obtain initial control points; A target control point is screened out from the initial control points according to a registration result obtained after registering the three-dimensional model with the first video frame to obtain the control point pair, wherein the target control point has a corresponding control point in the three-dimensional model.

5. The medical image processing system according to any one of claims 1 to 4, characterized in that: The processing module registers the three-dimensional model with the second video frame according to the corresponding relationship, including: Determining the relative position relationship between the target tissue and the intraoperative image sensor and the elastic deformation of each feature point of the three-dimensional model according to the corresponding relationship between the second video frame and the three-dimensional model, wherein the intraoperative image sensor is used to collect video images of the target tissue during surgery; The three-dimensional model and the second video frame are registered according to the elastic deformation and the relative position relationship.

6. The medical image processing system according to claim 5, characterized in that: The determining, according to the correspondence between the second video frame and the three-dimensional model, the relative position relationship between the target tissue and the intraoperative image sensor and the elastic deformation of each feature point of the three-dimensional model includes: Determining a relative position relationship between the target tissue and the intraoperative image sensor according to the corresponding relationship; Determining displacement change information of rigid motion of the three-dimensional model according to the relative position relationship; Determining a virtual force causing a change in the three-dimensional model according to the displacement change information; Simulating the elastic deformation of each feature point in the three-dimensional model according to the virtual force and the constraint conditions of the elastic deformation to obtain the elastic deformation of each feature point in the three-dimensional model, or; The elastic deformation of each characteristic point in the three-dimensional model is calculated using a finite element analysis method according to the virtual force and the biomechanical model of the target tissue.

7. A medical image processing method, characterized in that: include: Acquire video images of target tissue during surgery; The method comprises: determining a correspondence between the second video frame and the three-dimensional model according to a first video frame, the three-dimensional model of the target tissue, and a second video frame, and registering the three-dimensional model with the second video frame according to the correspondence, wherein the first video frame and the second video frame are respectively consecutive video frames, including: determining a vector field for pixel movement in the second video frame according to the first video frame and the second video frame; determining an updated position of a control point in the second video frame according to the vector field, and determining a correspondence between the second video frame and the three-dimensional model according to the updated position of the control point and the correspondence between the first video frame and the three-dimensional model; and Determining the update position of the control point in the second video frame according to the vector field includes: mapping the feature points in the feature point dense area in the first video frame to the second video frame according to the vector field, and determining the update position of the control point in the second video frame according to the weight value of the feature point relative to the control point, wherein the feature point dense area is an area in the second video frame where the number of feature points is greater than the number threshold; mapping the control points in the feature point sparse area in the first video frame to the second video frame according to the vector field, and obtaining the update position of the control point in the feature point sparse area, wherein the feature point sparse area is an area in the second video frame where the number of feature points is less than the number threshold; The registered second video frame and the three-dimensional model are displayed in real time.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to claim 7 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to claim 7 is implemented.

Images