Medical image processing method, system, electronic device and storage medium

Abstract

A medical image processing method, system, electronic device and storage medium provided by the embodiments of the present application, wherein a three-dimensional model and a two-dimensional image of a target tissue area are inputted through an input module; a processing module performs feature matching according to a first feature selected by a user in the three-dimensional model and the two-dimensional image; the processing module aligns the three-dimensional model and the two-dimensional image after feature matching based on the internal constraints of the three-dimensional model and the control feature projection constraint model, wherein the internal constraints of the three-dimensional model include inherent constraints of the target tissue, and dynamic alignment of the three-dimensional model and the two-dimensional image can be achieved, and the alignment accuracy can be improved.

Description

Technical Field

[0001] The present application belongs to the field of medical image processing technology, and in particular, relates to a medical image processing method, system, electronic device and storage medium. Background Art

[0002] In the medical field, aligning the preoperative 3D model to the endoscopic field of view can provide real-time guidance to doctors. Aligning the preoperative 3D model to the endoscopic field of view means aligning the 3D model with the target tissue on the 2D image so that the projection contour of the 3D model coincides with the contour of the target tissue on the 2D image. Due to many reasons such as the deformation of the actual tissue during surgery, the deformation features on the 2D image change, but the changes in the 3D model are not completely consistent with the changes in the shape features on the 2D image, and sometimes the difference is so large that dynamic and precise matching cannot be achieved. Summary of the invention

[0003] In response to the above problems, the embodiments of the present application provide a medical image processing method, system, electronic device and storage medium, which can realize dynamic registration of a three-dimensional model and a two-dimensional image.

[0004] The present application embodiment provides a medical image processing system, including:

[0005] An input module, used for inputting a three-dimensional model and a two-dimensional image of a target tissue area;

[0006] A processing module, used for performing feature matching between a first feature selected by a user in the three-dimensional model and a first feature selected in the two-dimensional image;

[0007] The processing module is also used to align the three-dimensional model and the two-dimensional image after feature matching based on the internal constraints of the three-dimensional model and the control feature projection constraint model, and the internal constraints of the three-dimensional model include inherent constraints of the target tissue.

[0008] In some embodiments, the control feature projection constraint model is used to minimize the reprojection error between the coordinate points of the feature points in the three-dimensional model and the coordinate points of the corresponding feature points in the two-dimensional image.

[0009] In some embodiments, the processing module is further used to register the three-dimensional model and the two-dimensional image after feature matching based on the internal constraints of the three-dimensional model and the control feature projection constraint model, including:

[0010] obtaining candidate coordinates of a first feature in the three-dimensional model that satisfies internal constraints of the three-dimensional model,

[0011] Performing control feature projection constraint model resolution on the candidate coordinates of the first feature of the three-dimensional model and the target coordinates of the first feature of the two-dimensional image, and calculating the target coordinates of the first feature of the three-dimensional model.

[0012] In some embodiments, the processing module is further configured to drive the movement of the three-dimensional model on the two-dimensional image according to the original coordinates and the target coordinates of the first feature of the three-dimensional model, so that the three-dimensional model is registered with the two-dimensional image.

[0013] In some embodiments, the medical imaging system further includes a display module, and the display module is configured to display the three-dimensional model and the two-dimensional image; the processing module performs feature matching on the first feature selected by the user in the three-dimensional model and the first feature selected by the user in the two-dimensional image, including:

[0014] In response to the first operation information of the user on the three-dimensional model, obtaining the first feature selected in the three-dimensional model;

[0015] In response to the second operation information of the user on the two-dimensional image, obtaining the first feature selected in the two-dimensional image;

[0016] Matching the first feature selected in the three-dimensional model with the first feature selected in the two-dimensional image.

[0017] In some embodiments, the processing module is configured to match the first feature of the three-dimensional model with the first feature of the two-dimensional image, including:

[0018] Determining the marking order of the first feature of the three-dimensional model based on the first operation information;

[0019] Determining the marking order of the first feature of the two-dimensional image based on the second operation information;

[0020] Matching the first feature of the three-dimensional model with the first feature of the two-dimensional image based on the marking order of the first feature of the three-dimensional model and the marking order of the first feature of the two-dimensional image.

[0021] In some embodiments, the display module is configured to display the three-dimensional model and the two-dimensional image, including:

[0022] Displaying the two-dimensional image in a first area and displaying the three-dimensional model in a second area, where the first area and the second area do not overlap, or superimposing and displaying the two-dimensional image and the three-dimensional model.

[0023] Another embodiment of the present application provides a medical image processing method, including:

[0024] Obtaining a three-dimensional model and a two-dimensional image of the target tissue region;

[0025] Performing feature matching on a first feature selected by a user in the three-dimensional model and a first feature selected in the two-dimensional image;

[0026] The feature-matched three-dimensional model and the two-dimensional image are registered with the three-dimensional model based on the internal constraints of the three-dimensional model and the control feature projection constraint model, wherein the internal constraints of the three-dimensional model include the inherent constraints of the target tissue.

[0027] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described methods when executing the computer program.

[0028] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned methods is implemented.

[0029] An embodiment of the present application provides a computer program product. When the computer program product is executed on a terminal device, the electronic device executes any one of the above methods.

[0030] A medical image processing system provided by an embodiment of the present application inputs a three-dimensional model and a two-dimensional image of a target tissue area through an input module; a processing module performs feature matching on a first feature selected by a user in the three-dimensional model and a first feature selected in the two-dimensional image; the processing module aligns the three-dimensional model and the two-dimensional image after feature matching based on internal constraints of the three-dimensional model and a control feature projection constraint model, wherein the internal constraints of the three-dimensional model include inherent constraints of the target tissue, and can realize dynamic alignment of the three-dimensional model and the two-dimensional image, and can improve the alignment accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Hereinafter, the present application will be described in more detail according to embodiments and with reference to the accompanying drawings.

[0032] Figure 1 A schematic diagram of the structure of a medical image processing system provided in an embodiment of the present application;

[0033] Figure 2 A schematic diagram of an implementation flow of a medical image processing method provided in an embodiment of the present application;

[0034] Figure 3 A schematic diagram of an implementation flow of a medical image processing method provided in an embodiment of the present application;

[0035] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0036] In the drawings, the same reference numerals are used for the same components, and the drawings are not drawn to scale. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0038] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] If similar descriptions of "first\second\third" appear in the application documents, the following instructions are added. In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0041] Based on the problems in the related art, an embodiment of the present application provides a medical image processing system. An embodiment of the present application provides a medical image processing system. The modules included in the system and the units included in each module can be implemented by a processor in a computer device; of course, they can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU, Central Processing Unit), a microprocessor (MPU, Micro Processor Unit), a digital signal processor (DSP, Digital Signal Processing) or a field programmable gate array (FPGA, Field Programmable Gate Array), etc.

[0042] The present application embodiment provides a medical image processing system. Figure 1A schematic diagram of the structure of a medical image processing system provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the medical image processing system 100 includes: an input module 101 and a processing module 102, the input module is used to perform feature matching between a first feature selected by a user in the three-dimensional model and a first feature selected in the two-dimensional image; the processing module is also used to perform feature matching based on the first feature selected by the user in the three-dimensional model and the two-dimensional image; the processing module is also used to align the three-dimensional model and the two-dimensional image after feature matching based on the internal constraints of the three-dimensional model and the control feature projection constraint model, wherein the internal constraints of the three-dimensional model include inherent constraints of the target tissue.

[0043] In the embodiment of the present application, the target tissue may be an organ in the human body, such as the liver, heart, stomach, etc. The inherent constraints of the target tissue refer to the inherent limitations of a specific organ in the body in terms of structure and material.

[0044] In the embodiment of the present application, the two-dimensional image may be an image during surgery. During the operation, the surgeon may use the image acquisition module to acquire an image of the target tissue during surgery. The three-dimensional model may include: a three-dimensional model of the target tissue established before surgery.

[0045] In the embodiment of the present application, the selected first feature may be: a feature point, a feature line, etc. A feature point may also be called a control point, and a feature line may also be called a control line.

[0046] In the embodiment of the present application, the user can select the first feature in the three-dimensional model and the two-dimensional image through an interactive interface.

[0047] In the embodiment of the present application, the selected first feature exists in both the 3D model and the 2D image. The processing module can perform feature matching based on the selected first feature existing in both the 3D model and the 2D image, thereby achieving initial registration of the 3D model and the 2D image.

[0048] In an embodiment of the present application, the processing module performs feature matching on the first feature selected by the user in the three-dimensional model and the first feature selected in the two-dimensional image, which may be to optimize the displacement difference between the first features in the three-dimensional model and the two-dimensional image, so that the first feature in the three-dimensional model approaches the first feature in the two-dimensional image.

[0049] In the embodiment of the present application, the internal constraints of the three-dimensional model can be a physical simulation model, and the internal constraints of the three-dimensional model can be characterized by a model based on the finite element method, a position dynamics model, a mass spring model, etc. Taking the model based on the finite element method as an example, the internal constraints of the three-dimensional model can be characterized by Young's modulus and Poisson's ratio to characterize the properties of materials of different tissues. The elastic model can be established by Young's modulus, and then the calculation relationship between the elastic model, Poisson's ratio and the node coordinates in the three-dimensional model can be constructed. The calculation relationship can be expressed as:

[0050] ;

[0051] Among them, E is the elastic model, v is Poisson's ratio, and u is the node coordinate of the model. The model is the constitutive equation of the material.

[0052] In an embodiment of the present application, the feature projection constraint model is controlled so that the reprojection error between the coordinate points of the feature points in the three-dimensional model and the coordinate points of the corresponding feature points in the two-dimensional image is minimized. A calculation formula for minimizing the reprojection error between the coordinate points of the three-dimensional model and the coordinate points in the two-dimensional image can be established based on the pose matrix of the model in the camera coordinate system and the camera intrinsic parameter matrix, thereby obtaining the controlled feature projection constraint model.

[0053] The control feature projection constraint module can be expressed as:

[0054] ;

[0055] Among them, u 3d is the feature point on the 3D model, u 2d are feature points on the two-dimensional image. T is the pose matrix of the model in the camera coordinate system, and K is the camera intrinsic parameter matrix.

[0056] In an embodiment of the present application, the control feature projection constraint model uses the reprojection error to construct a loss function, and then minimizes the loss function by solving the displacement of the feature points through a nonlinear optimization method.

[0057] In an embodiment of the present application, the coordinates of the first feature in the three-dimensional model can be solved by the internal constraints of the three-dimensional model, and then the coordinates of the first feature can be optimized by controlling the feature projection constraint model to obtain the target coordinates of each feature point. When the target coordinates of each feature simultaneously make the internal constraints of the three-dimensional model and the control feature projection constraint model hold true, it is determined that each feature has completed the registration, thereby realizing the registration of the three-dimensional model and the two-dimensional image.

[0058] In the embodiment of the present application, when solving the feature projection constraint model by controlling the feature point, the feature points on the three-dimensional model are subjected to the rigid transformation of the pose matrix T and the projection transformation of the camera intrinsic parameter matrix K, and then obtain the 2D coordinates on the two-dimensional image coordinate system. The solution of the controlled feature projection constraint model optimizes the displacement of the feature points in the three-dimensional model by optimizing the gap between the projection point of the feature point in the three-dimensional model and the coordinate point on the two-dimensional image, thereby constraining the feature points in the three-dimensional model, which is similar to a driving force that drives the features of the three-dimensional model to deform, thereby realizing the process of constraining the deformation of the features in the three-dimensional model.

[0059] In some embodiments, the processing module is used to align the three-dimensional model and the two-dimensional image after feature matching based on the internal constraints of the three-dimensional model and the control feature projection constraint model, including: obtaining candidate coordinates of the first feature in the three-dimensional model that satisfies the internal constraints of the three-dimensional model, solving the control feature projection constraint model according to the candidate coordinates of the first feature of the three-dimensional model and the target coordinates of the first feature of the two-dimensional image, and calculating the target coordinates of the first feature of the three-dimensional model.

[0060] In the embodiment of the present application, the candidate coordinates are coordinates that satisfy the internal constraint model of the three-dimensional model, and the target coordinates are coordinates that satisfy both the internal constraint of the three-dimensional model and the control feature projection constraint model.

[0061] In an embodiment of the present application, the processing module is also used to drive the movement of the three-dimensional model on the two-dimensional image according to the original coordinates and target coordinates of the first feature of the three-dimensional model, so that the three-dimensional model is aligned with the two-dimensional image.

[0062] In some embodiments, the processing module may couple the internal constraints of the three-dimensional model and the control feature projection constraint model to obtain a coupled model; determine the target coordinates of each feature in the 3D model based on the coupled model; and align the three-dimensional model with the two-dimensional image based on the target coordinates.

[0063] In an embodiment of the present application, the processing module can use the Lagrangian method to couple the internal constraint model of the three-dimensional model and the control feature projection constraint model to obtain a coupling model. By analyzing the coupling model, the coordinates of the feature points in the two-dimensional image coordinate system can be obtained, thereby completing the alignment of the feature points. By solving each feature point, the alignment of the three-dimensional model and the two-dimensional image can be achieved.

[0064] The coupling model can be expressed as:

[0065] ;

[0066] By analyzing the coupling model, the coordinates of the feature point u in the two-dimensional image coordinate system can be obtained, so that the registration of the feature point u can be completed. By solving each feature point, the registration of the three-dimensional model and the two-dimensional image can be achieved.

[0067] The medical image processing system provided in the embodiment of the present application inputs a three-dimensional model and a two-dimensional image of a target tissue area through an input module; the processing module performs feature matching according to a first feature selected by a user in the three-dimensional model and the two-dimensional image; the processing module aligns the three-dimensional model and the two-dimensional image after feature matching based on the internal constraints of the three-dimensional model and the control feature projection constraint model, wherein the internal constraints of the three-dimensional model include inherent constraints of the target tissue, and can drive the three-dimensional model and the two-dimensional image to perform initial alignment based on the selected features, and the alignment through the internal constraints of the three-dimensional model and the control feature projection constraint model can improve the alignment accuracy.

[0068] In some embodiments, the medical image processing system also includes: a display module, which is used to display the three-dimensional model and the two-dimensional image. After the input module inputs the three-dimensional model and the two-dimensional image of the target tissue area, the display module can display the three-dimensional model and the two-dimensional image of the target tissue area.

[0069] In an embodiment of the present application, during display, the two-dimensional image can be displayed in a first area and the three-dimensional model can be displayed in a second area, wherein the first area and the second area do not overlap, for example, the first area is the left area and the second area is the right area.

[0070] In some embodiments, when displaying, the three-dimensional model and the two-dimensional image may be displayed in a superimposed manner. When displaying in a superimposed manner, the three-dimensional model may be set to a preset transparency.

[0071] In the embodiment of the present application, the three-dimensional model and the two-dimensional image of the target tissue area input by the input module are displayed by the display module, which can facilitate the user to select the first feature.

[0072] In some embodiments, the processing module performs feature matching on a first feature selected by a user in the three-dimensional model and a first feature selected in the two-dimensional image, including: obtaining the first feature selected in the three-dimensional model in response to first operation information of the user on the three-dimensional model; obtaining the first feature selected in the two-dimensional image in response to second operation information of the user on the two-dimensional image; and matching the first feature selected in the three-dimensional model and the first feature selected in the two-dimensional image.

[0073] In an embodiment of the present application, the user can operate the three-dimensional model through various interactive methods such as a mouse, a handle, a touch screen, gesture recognition, etc., so that the processing module obtains the user's first operation information on the three-dimensional model. The first operation information may include: clicking, drawing, etc.

[0074] For example, you can tap with one finger on the touch screen, draw with two fingers, or click and drag on the touch screen to tap / draw, or click the mouse to tap, hold down and drag to draw, or click different buttons to enter the tap / draw mode. Different gestures correspond to different operations.

[0075] In the embodiment of the present application, the first feature may be a feature point or a feature line, etc.

[0076] In the embodiment of the present application, the two-dimensional image can be operated through various interactive methods such as a mouse, a handle, a touch screen, gesture recognition, etc., so that the processing module obtains the second operation information of the user on the two-dimensional image.

[0077] In the embodiment of the present application, matching can be performed based on the feature information of the first feature, so as to match the first feature of the three-dimensional model with the first feature of the two-dimensional image.

[0078] In the embodiment of the present application, the user selects the first feature to drive the feature of the model to shift, thereby achieving preliminary alignment between the model and the image.

[0079] In some embodiments, the processing module matches the first feature of the three-dimensional model with the first feature of the two-dimensional image, including: determining the marking order of the first feature of the three-dimensional model based on the first operation information; determining the marking order of the first feature of the two-dimensional image based on the second operation information; matching the first feature of the three-dimensional model with the first feature of the two-dimensional image based on the marking order of the first feature of the three-dimensional model and the marking order of the first feature of the two-dimensional image.

[0080] In the embodiment of the present application, since the first operation information is implemented through various interactive methods such as a mouse, a handle, a touch screen, and gesture recognition, the processing module can identify the marking order of the first feature, and similarly, the marking order of the first feature of the two-dimensional image can also be identified. When matching the first feature of the three-dimensional model with the first feature of the two-dimensional image, it can be considered that the three-dimensional model and the first feature of the two-dimensional image are corresponding, and computer vision algorithms or geometric registration techniques can be used to identify matching feature points, feature descriptors, or feature vectors, thereby associating the first feature of the three-dimensional model with the first feature of the two-dimensional image.

[0081] In some embodiments, the first feature may be a feature point, and the processing module matches the first feature of the three-dimensional model with the first feature of the two-dimensional image, including: using a permutation and combination algorithm to generate a point pair combination of the first feature of the three-dimensional model and the first feature of the two-dimensional image; calculating a matching score for each point pair combination; and determining a target matching result based on the matching score to match the first feature of the three-dimensional model with the first feature of the two-dimensional image.

[0082] In an embodiment of the present application, a permutation and combination algorithm can be used to generate a point pair combination consisting of a first feature of the three-dimensional model and a first feature of the two-dimensional image. The first feature of the three-dimensional model and the first feature of the two-dimensional image are combined using the permutation and combination algorithm to form all possible point pairs. A feature matching algorithm or a similarity metric is then used to compare the similarities between the point pairs to determine the degree of matching between them. Based on the calculated matching score, the point pair combination with the highest matching score is selected as the target matching result to complete the matching of the first feature of the three-dimensional model and the first feature of the two-dimensional image.

[0083] In some embodiments, the first feature may be a feature line, and the processing module matches the first feature of the three-dimensional model with the first feature of the two-dimensional image, including: determining a first shape of the first feature of the three-dimensional model, and determining a feature description of each point on the first feature of the three-dimensional model based on the first shape; determining a second shape of the first feature of the two-dimensional image, and determining a feature description of each point on the first feature of the two-dimensional image based on the second shape; determining the feature description of each point on the first feature of the three-dimensional model as a first vector in a feature space, and determining the feature description of each point on the first feature of the two-dimensional image as a second vector in the feature space; based on the first vector and the second vector, matching the first feature of the three-dimensional model with the first feature of the two-dimensional image.

[0084] In the embodiment of the present application, it is first necessary to extract feature points or feature lines, which can represent the key features of the shape. In a three-dimensional model, a shape descriptor or a geometric feature can be used to describe each feature point to determine the feature description of each point on the first feature of the three-dimensional model; in a two-dimensional image, an image feature descriptor (such as SIFT, SURF, etc.) can be used to describe each feature point to determine the feature description of each point on the first feature of the two-dimensional image. For each feature point, its feature description is generated, and these descriptions can be local descriptors that describe the local feature information around the feature point. These descriptions can be used to compare and match feature points. The feature description of each feature point is converted into a feature vector to construct a feature space. For three-dimensional models and two-dimensional images, the feature vectors are combined into a first vector and a second vector. Use a similarity measurement method (such as nearest neighbor matching, RANSAC, etc.) to compare the first vector and the second vector to find the best matching correspondence. The corresponding relationship between the first feature of the three-dimensional model and the first feature of the two-dimensional image can be determined by matching.

[0085] In some embodiments, the first feature may be a feature line, and the processing module matches the first feature of the three-dimensional model with the first feature of the two-dimensional image, including: determining the nearest point on the first feature of the three-dimensional model to the first feature of the two-dimensional image, and forming a point pair with the point on the first feature of the three-dimensional model and the nearest point on the first feature of the two-dimensional image; determining a transformation matrix based on the point pair; transforming the first feature of the three-dimensional model or the first feature of the two-dimensional image based on the transformation matrix to obtain a transformed control line; performing iterative calculation based on the changed control line to update the transformation matrix; and matching the first feature of the three-dimensional model with the first feature of the two-dimensional image when the transformation matrix converges.

[0086] In the embodiment of the present application, a nearest neighbor algorithm can be used to find the nearest points of the first feature of the three-dimensional model on the first feature of the two-dimensional image, and form them into point pairs. The transformation matrix can be estimated using a minimum mean square error, such as a least squares method.

[0087] In the embodiment of the present application, the changed control line can be iteratively calculated, and the transformation matrix can be updated by an optimization algorithm until convergence, and the optimization algorithm can be a gradient descent algorithm. When the transformation matrix converges, the first feature of the three-dimensional model and the first feature of the two-dimensional image can be matched to determine the corresponding relationship between them.

[0088] In some embodiments, the display module is further used to overlay and display the registered two-dimensional image and the three-dimensional model.

[0089] Based on the aforementioned medical image processing system, an embodiment of the present application provides a medical image processing method. The embodiment of the present application provides a medical image processing method that can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), medical imaging devices, etc. The embodiment of the present application does not impose any restrictions on the specific types of electronic devices.

[0090] The functions implemented by the medical image processing method provided in the embodiment of the present application can be implemented by calling program codes by a processor of an electronic device, wherein the program codes can be stored in a computer storage medium.

[0091] The present application embodiment provides a medical image processing method. Figure 2 A schematic diagram of the implementation flow of a medical image processing method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, including:

[0092] Step S201, obtaining a three-dimensional model and a two-dimensional image of the target tissue area;

[0093] Step S202, performing feature matching between a first feature selected by a user in the three-dimensional model and a first feature selected in the two-dimensional image;

[0094] Step S203 , registering the feature-matched three-dimensional model and the two-dimensional image with the three-dimensional model based on the internal constraints of the three-dimensional model and the control feature projection constraint model.

[0095] In some embodiments, the control feature projection constraint model is used to minimize the reprojection error between the coordinate points of the feature points in the three-dimensional model and the coordinate points of the corresponding feature points in the two-dimensional image.

[0096] In some embodiments, registering the feature-matched three-dimensional model and the two-dimensional image based on the internal constraints of the three-dimensional model and the control feature projection constraint model includes:

[0097] Obtain candidate coordinates of the first feature in the 3D model that satisfy the internal constraints of the 3D model,

[0098] The control feature projection constraint model is solved according to the candidate coordinates of the first feature of the three-dimensional model and the target coordinates of the first feature of the two-dimensional image, and the target coordinates of the first feature of the three-dimensional model are calculated.

[0099] In some embodiments, the method further comprises:

[0100] The three-dimensional model is driven to move on the two-dimensional image according to the original coordinates and the target coordinates of the first feature of the three-dimensional model, so that the three-dimensional model is registered with the two-dimensional image.

[0101] In some embodiments, the method further comprises:

[0102] Displaying the three-dimensional model and the two-dimensional image, and performing feature matching on a first feature selected by a user in the three-dimensional model and a first feature selected by a user in the two-dimensional image, comprises:

[0103] In response to first operation information of a user on the three-dimensional model, obtaining a first feature selected in the three-dimensional model;

[0104] In response to second operation information of the user on the two-dimensional image, obtaining a first feature selected in the two-dimensional image;

[0105] The first feature selected in the three-dimensional model is matched with the first feature selected in the two-dimensional image.

[0106] In some embodiments, matching the first feature of the three-dimensional model with the first feature of the two-dimensional image includes:

[0107] Determining a marking order of a first feature of the three-dimensional model based on the first operation information;

[0108] determining a marking order of the first feature of the two-dimensional image based on the second operation information;

[0109] The first feature of the three-dimensional model and the first feature of the two-dimensional image are matched based on the marking order of the first feature of the three-dimensional model and the marking order of the first feature of the two-dimensional image.

[0110] In some embodiments, displaying the three-dimensional model and the two-dimensional image includes:

[0111] The two-dimensional image is displayed in a first area, and the three-dimensional model is displayed in a second area, wherein the first area and the second area do not overlap, or the two-dimensional image and the three-dimensional model are displayed in a superimposed manner.

[0112] The working process of the above method can refer to the corresponding process in the aforementioned system embodiment, which will not be repeated here.

[0113] Based on the aforementioned medical image processing method, the present application embodiment further provides a medical image processing method. Figure 3A schematic diagram of the implementation flow of a medical image processing method provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, including:

[0114] Step S301, obtaining user input.

[0115] In the embodiments of the present application, the user may input information through various interactive methods, thereby obtaining the user's input, for example, inputting information through various interactive methods such as a mouse, a handle, a touch screen, and gesture recognition.

[0116] Step S302: performing two-dimensional image interaction operations and three-dimensional model interaction operations recognition based on user input.

[0117] In an embodiment of the present application, a two-dimensional image and a three-dimensional model can be displayed on a display screen. When displayed, the display can be split left and right. In some embodiments, the three-dimensional model and the two-dimensional image can also be superimposed on the display interface, and the three-dimensional model is set to be semi-transparent. The interaction between the two-dimensional image and the three-dimensional model is distinguished by buttons or left and right mouse buttons. Under this operation, it is easier to correspond the characteristic points and lines of the model.

[0118] Step S303: identifying control points and control lines.

[0119] In the embodiment of the present application, control points and control lines can be partitioned through user input. For example, the user taps with one finger, draws with two fingers, taps with a single click of the mouse, presses and drags to draw a line, etc. The control points can be considered as feature points, and the control lines can be considered as feature lines.

[0120] Step S304: feature matching.

[0121] In the embodiment of the present application, in the case of control points, point matching can be performed, and the point matching can be matched according to the order of point marking. It is also possible to traverse the results of the permutation and combination through an algorithm and select the best matching point pair using the matching results. In this case, the order of point marking does not need to be guaranteed.

[0122] In the case of a control line, the line matching method can be used for matching. The feature description of the points on the line can be calculated based on the shape of the line, and then the matching can be performed in the feature space. The matching point pairs can be obtained by iteratively calculating the nearest point pairs, and then the two-dimensional projection matrix can be calculated. The curve is transformed by the two-dimensional projection matrix, and the result of the transformation is updated as the input of the next iteration until convergence, completing the matching of the points on the line.

[0123] Step S305: registering the three-dimensional model and the two-dimensional image.

[0124] In the embodiment of the present application, the solution of the internal constraints of the three-dimensional model and the solution of the control feature projection constraint model can adopt the Gauss-Seidel method, and the internal constraints of the three-dimensional model and the feature projection constraints are solved independently in one iteration process. In addition, the internal constraints of the three-dimensional model and the control feature projection constraint model can be coupled and solved by using the Lagrange multiplier method.

[0125] The method provided in the embodiment of the present application uses the user's interactive input as the driver for the three-dimensional model registration, driving the three-dimensional model to displace and deform, making the entire registration process intuitive and controllable, in line with the user's intuition. Through user interaction, control points / lines are generated on the two-dimensional image, and different types of marks can be automatically identified. At the same time, during the interaction process, the user can gradually apply control point / line constraints, interact while registering, and can also add enough control point / line constraints at a time to directly obtain the final registration result.

[0126] The method provided in the embodiment of the present application makes the registration result more accurate by calculating the rigid displacement and the deformation displacement. The method provided in the embodiment of the present application can be compatible with data of two-dimensional images and three-dimensional models of any format and source. At the same time, the stiffness parameter of the deformation model is a user-adjustable mode. By the user modifying the stiffness parameter of the physical model, it can be compatible with tissues with different mechanical properties, such as rigid bones and soft tissue livers.

[0127] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 4 As shown, the electronic device 3 of this embodiment may include: at least one processor 30 ( Figure 4 Only one processor 30 is shown in the figure), a memory 31, and a computer program 32 stored in the memory 31 and executable on at least one processor 30. When the processor 30 executes the computer program 32, the steps in any of the above-mentioned method embodiments are implemented, for example Figure 2 Steps S201 to S203 in the illustrated embodiment. Alternatively, when the processor 30 executes the computer program 32, the functions of each module / unit in the above-mentioned system embodiments are implemented, for example Figure 1 The functions of modules 101 to 102 are shown.

[0128] Exemplarily, the computer program 32 may be divided into one or more modules / units, one or more modules / units are stored in the memory 31, and are executed by the processor 30 to complete the present application. One or more modules / units may be a series of computer program 32 instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 32 in the electronic device 3.

[0129] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program 32. When the computer program 32 is executed by the processor 30, the steps in the above-mentioned method embodiments can be implemented.

[0130] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0131] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. According to this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program 32, and the computer program 32 can be stored in a computer-readable storage medium. When the computer program 32 is executed by the processor 30, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program 32 includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include at least: any entity or device capable of carrying the computer program code to the terminal, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0132] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0133] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0134] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0135] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A medical image processing system, characterized in that: include: An input module, used for inputting a three-dimensional model and a two-dimensional image of a target tissue area; A processing module, used for performing feature registration between a first feature selected by a user in the three-dimensional model and a first feature selected in the two-dimensional image; The processing module is also used to align the three-dimensional model and the two-dimensional image after feature alignment based on the internal constraints of the three-dimensional model and the control feature projection constraint model, including: using the Lagrangian method to couple the internal constraint model of the three-dimensional model and the control feature projection constraint model to obtain a coupling model, and by analyzing the coupling model, the coordinates of the feature points in the three-dimensional model in the two-dimensional image coordinate system are obtained, thereby completing the alignment of the feature points. The internal constraints of the three-dimensional model include the inherent constraints of the target tissue.

2. The medical image processing system according to claim 1, characterized in that: The control feature projection constraint model is used to minimize the reprojection error between the coordinate points of the feature points in the three-dimensional model and the coordinate points of the corresponding feature points in the two-dimensional image.

3. The medical image processing system according to claim 1 or 2, characterized in that: The processing module is also used to register the three-dimensional model and the two-dimensional image after feature registration based on the internal constraints of the three-dimensional model and the control feature projection constraint model, including: obtaining candidate coordinates of a first feature in the three-dimensional model that satisfies internal constraints of the three-dimensional model, A control feature projection constraint model is solved according to the candidate coordinates of the first feature of the three-dimensional model and the target coordinates of the first feature of the two-dimensional image to calculate the target coordinates of the first feature of the three-dimensional model.

4. The medical image processing system according to claim 3, characterized in that: The processing module is also used to drive the movement of the three-dimensional model on the two-dimensional image according to the original coordinates and the target coordinates of the first feature of the three-dimensional model, so that the three-dimensional model is aligned with the two-dimensional image.

5. The medical image processing system according to claim 1, characterized in that: The medical image processing system further includes a display module, which is used to display the three-dimensional model and the two-dimensional image; the processing module performs feature matching on a first feature selected by a user in the three-dimensional model and a first feature selected by a user in the two-dimensional image, including: In response to first operation information of a user on the three-dimensional model, obtaining a first feature selected in the three-dimensional model; In response to second operation information of the user on the two-dimensional image, obtaining a first feature selected in the two-dimensional image; The first feature selected in the three-dimensional model and the first feature selected in the two-dimensional image are registered.

6. The medical image processing system according to claim 5, characterized in that: The processing module is used to register the first feature of the three-dimensional model with the first feature of the two-dimensional image, including: Determining a marking order of a first feature of the three-dimensional model based on the first operation information; determining a marking order of the first feature of the two-dimensional image based on the second operation information; The first feature of the three-dimensional model and the first feature of the two-dimensional image are registered based on the marking order of the first feature of the three-dimensional model and the marking order of the first feature of the two-dimensional image.

7. The medical image processing system according to claim 5, characterized in that: The display module is used to display the three-dimensional model and the two-dimensional image, and includes: The two-dimensional image is displayed in a first area, and the three-dimensional model is displayed in a second area, wherein the first area and the second area do not overlap, or the two-dimensional image and the three-dimensional model are displayed in a superimposed manner.

8. A medical image processing method, characterized in that: include: Obtaining a three-dimensional model and a two-dimensional image of the target tissue region; Performing feature registration on a first feature selected by a user in the three-dimensional model and a first feature selected in the two-dimensional image; The three-dimensional model and the two-dimensional image after feature registration are registered based on the internal constraints of the three-dimensional model and the control feature projection constraint model, including: using the Lagrangian method to couple the internal constraint model of the three-dimensional model and the control feature projection constraint model to obtain a coupling model, and by analyzing the coupling model, the coordinates of the feature points in the three-dimensional model in the two-dimensional image coordinate system are obtained, thereby completing the registration of the feature points. The internal constraints of the three-dimensional model include the inherent constraints of the target tissue.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the medical image processing method as claimed in claim 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the medical image processing method as claimed in claim 8 is implemented.

Images