Puncture positioning method and device, electronic device and storage medium

Abstract

This application discloses a puncture localization method and apparatus, electronic device, and storage medium. The method includes: acquiring a preoperative image based on intraoperative images, wherein the intraoperative images are composed of a portion of the preoperative images; generating a simulated puncture needle in the coordinate system of the preoperative images based on a puncture needle in the coordinate system of a navigation device, wherein the coordinate system of the preoperative images is the same as the coordinate system of the intraoperative images; and sending the simulated puncture needle located in the coordinate system of the intraoperative images to a front-end device to facilitate positioning of the simulated puncture needle by the front-end device, thereby reducing the number of times images need to be taken of the patient during the operation and thus avoiding multiple radiation exposures to the patient.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a puncture positioning method and device, an electronic device and a storage medium. Background Art

[0002] Pathological diagnosis of lesion tissue plays an important guiding role in disease diagnosis. Clinically, puncture surgery is usually performed under the guidance of CT (Computed Tomography) images to obtain pathological specimens. However, in order to accurately locate the target area during percutaneous puncture surgery, it is often necessary to take multiple intraoperative images of the patient during the operation, which not only makes the operation process too complicated, but also causes multiple radiation to the patient. Summary of the invention

[0003] In view of this, the embodiments of the present application are dedicated to providing a puncture positioning method and device, an electronic device and a storage medium, which can reduce the number of times the patient is imaged during surgery, thereby avoiding multiple radiation exposure to the patient.

[0004] According to a first aspect of an embodiment of the present application, a puncture positioning method is provided, comprising: acquiring a preoperative image based on an intraoperative image, wherein the intraoperative image is composed of part of the preoperative image; generating a simulated puncture needle in the coordinate system of the preoperative image based on the puncture needle in the coordinate system of a navigation device, wherein the coordinate system of the preoperative image is the same as the coordinate system of the intraoperative image; and sending the simulated puncture needle located in the coordinate system of the intraoperative image to a front-end device, so as to facilitate positioning of the simulated puncture needle at the front-end device.

[0005] In some embodiments, the method further comprises: receiving the intraoperative image sent by an image archiving and communication system.

[0006] In some embodiments, obtaining the preoperative image based on the intraoperative image includes: analyzing the intraoperative image to obtain patient information of the intraoperative image; sending the patient information to the image archiving and communication system to query whether the preoperative image exists; and when the preoperative image exists, extracting the preoperative image from the image archiving and communication system.

[0007] In some embodiments, analyzing the intraoperative images to obtain patient information of the intraoperative images includes: performing data screening on the intraoperative images according to preset configuration rules; and performing patient information identification on the screened intraoperative images to obtain patient information of the intraoperative images.

[0008] In some embodiments, the patient information includes patient name, patient number, patient gender and / or date of birth.

[0009] In some embodiments, the method further includes: obtaining segmentation results of tissues and organs in the intraoperative images through a network model based on the intraoperative images; performing three-dimensional reconstruction on the segmentation results to obtain three-dimensional modeling results of the tissues and organs; and sending the three-dimensional modeling results to the front-end device to locate the position of the simulated puncture needle in the three-dimensional modeling results.

[0010] In some embodiments, the puncture needle in the coordinate system of the navigation device generates a simulated puncture needle in the coordinate system of the preoperative image, including: converting the coordinate system of the navigation device with the coordinate system of the marker on the preoperative image to obtain a first transformation matrix; converting the coordinate system of the marker on the preoperative image with the coordinate system of the preoperative image to obtain a second transformation matrix; and transforming the puncture needle into the coordinate system of the preoperative image according to the first transformation matrix and the second transformation matrix to generate the simulated puncture needle on the preoperative image.

[0011] In some embodiments, the coordinate system of the marker on the preoperative image and the coordinate system of the preoperative image are converted to obtain a second transformation matrix, including: performing a translation transformation on the key points of the marker on the preoperative image and the corresponding key points of the standard segmentation; obtaining a rotation transformation matrix between the segmentation result of the marker and the standard segmentation according to the registration model; determining the coordinate values ​​corresponding to the preset points in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image according to the rotation transformation matrix; determining the directions of the X, Y and Z axes of the coordinate system of the preoperative image relative to the X, Y and Z axes of the coordinate system of the marker on the preoperative image according to the coordinate values ​​corresponding to the preset points in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image; and determining the second transformation matrix according to the directions.

[0012] In some embodiments, the registration model includes a plurality of cascaded spatial transformation networks, the rotation transformation matrix includes a plurality of rotation transformation matrices, one spatial transformation network corresponds to one rotation transformation matrix, wherein, according to the rotation transformation matrix, the coordinate value corresponding to the preset point in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image is determined, including: calculating the intersection and union ratio of the input and output of each spatial transformation network in the plurality of spatial transformation networks; multiplying the rotation transformation matrix corresponding to the spatial transformation network whose intersection and union ratio is less than a preset threshold value by the coordinate value of the preset point in the coordinate system of the preoperative image, so as to obtain the coordinate value corresponding to the preset point in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image.

[0013] In some embodiments, the preset points include midpoints, edge points and / or axis points of the markers.

[0014] According to a second aspect of an embodiment of the present application, a puncture positioning device is provided, comprising: an acquisition module, configured to acquire a preoperative image based on an intraoperative image, wherein the intraoperative image is composed of part of the preoperative image; a generation module, configured to generate a simulated puncture needle in the coordinate system of the preoperative image based on the puncture needle in the coordinate system of a navigation device, wherein the coordinate system of the preoperative image is the same as the coordinate system of the intraoperative image; and a first sending module, configured to send the simulated puncture needle located in the coordinate system of the intraoperative image to a front-end device, so as to facilitate positioning of the simulated puncture needle at the front-end device.

[0015] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; and the processor is used to execute the method described in any one of the above embodiments.

[0016] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the method described in any of the above embodiments.

[0017] A puncture positioning method provided in an embodiment of the present application first obtains a preoperative image based on the intraoperative image, then generates a simulated puncture needle in the coordinate system of the preoperative image based on the puncture needle in the coordinate system of the navigation device, and finally sends the simulated puncture needle in the coordinate system of the intraoperative image to a front-end device, so as to facilitate the positioning of the simulated puncture needle at the front-end device, which can reduce the number of times the patient is imaged during surgery, thereby avoiding multiple radiation exposure to the patient. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0019] Figure 1 Shown is a schematic diagram of an implementation environment provided by an embodiment of the present application.

[0020] Figure 2 Shown is a schematic flow chart of a puncture positioning method provided in one embodiment of the present application.

[0021] Figure 3Shown is a schematic diagram of an intraoperative image provided by an embodiment of the present application.

[0022] Figure 4 Shown is a schematic diagram of a preoperative image provided by an embodiment of the present application.

[0023] Figure 5 Shown is a schematic flow chart of a puncture positioning method provided in another embodiment of the present application.

[0024] Figure 6 Shown is a schematic flow chart of a puncture positioning method provided in yet another embodiment of the present application.

[0025] Figure 7 Shown is a schematic flow chart of a puncture positioning method provided in yet another embodiment of the present application.

[0026] Figure 8 Shown is a schematic diagram of a coordinate system conversion process provided by an embodiment of the present application.

[0027] Fig. 9 Shown is a block diagram of a puncture positioning device provided in one embodiment of the present application.

[0028] Fig.10 Shown is a structural block diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] Application Overview

[0031] Pathological diagnosis of lesion tissue plays an important guiding role in disease diagnosis. Clinically, puncture surgery is usually performed under the guidance of lung CT (Computed Tomography) images to obtain pathological specimens for lesion diagnosis. During the puncture surgery, the doctor needs to determine the target lesion and the puncture path.

[0032] In order to accurately locate the target area during percutaneous puncture surgery, it is often necessary to take multiple intraoperative images of the patient during the operation, which not only makes the operation process too complicated, but also causes multiple radiation exposure to the patient.

[0033] Therefore, the embodiment of the present application combines intraoperative images (local images) with preoperative images (global images). By taking an intraoperative image once during the operation, accurate positioning of the target area can be achieved. At the same time, a simulated puncture needle can be generated in the intraoperative image to achieve puncture positioning in the intraoperative image.

[0034] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0035] Exemplary Systems

[0036] Figure 1 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application. The implementation environment includes an image archiving and communication system 110, a data acquisition module 120, a data storage center 130, a prediction module 140 and a front-end device 150. The data acquisition module 120 also includes an information pulling unit 121, a preprocessing unit 122 and a data sending unit 123.

[0037] The information pulling unit 121 can receive the intraoperative images sent by the image archiving and communication system 110. The preprocessing unit 122 performs data screening on the intraoperative images according to the preset configuration rules, and then performs patient information recognition on the screened intraoperative images to obtain the patient information of the intraoperative images, and finally sends the patient information to the image archiving and communication system 110 to query whether there are preoperative images. When there are preoperative images, the preoperative images are extracted from the image archiving and communication system 110. The data sending unit 123 sends the preoperative images to the data storage center 130. The data storage center 130 sends the preoperative images and the prediction tasks corresponding to the preoperative images to the prediction module 140. The prediction module 140 implements the prediction tasks on the preoperative images to obtain prediction results, and finally sends the prediction results to the data storage center 130. After obtaining the prediction results, the data storage center 130 sends the prediction results to the front-end device 150.

[0038] In addition, the data sending unit 123 can also send the intraoperative images to the data storage center 130 , and the data storage center 130 combines the intraoperative images with the prediction results and sends them together to the front-end device 150 .

[0039] In one embodiment, the front-end device 150 may refer to a user terminal or a server front-end device. When the front-end device 150 refers to a user terminal, medical staff can directly view intraoperative images and prediction results. When the front-end device 150 refers to a server front-end device, medical staff can access the server front-end device by entering the corresponding URL in the browser to view intraoperative images and prediction results.

[0040] In one embodiment, at least one network model may be deployed in the prediction module 140 to segment different tissues and organs in the preoperative image, detect lesions in the preoperative image, and align the preoperative image with the navigation device, etc., so as to obtain a prediction result corresponding to the intraoperative image. However, it should be noted that the embodiment of the present application does not specifically limit the type and number of network models in the prediction module 140, and those skilled in the art may make different choices according to actual needs.

[0041] Exemplary Methods

[0042] Figure 2 It is a flowchart of a puncture positioning method provided in one embodiment of the present application. Figure 2 The method described is performed by a computing device (e.g., a server), but the embodiments of the present application are not limited thereto. The server may be a single server, or may be composed of a plurality of servers, or may be a virtualization platform, or may be a cloud computing service center, and the embodiments of the present application are not limited thereto. Figure 2 As shown, the method includes the following contents.

[0043] S210: Acquire a preoperative image based on the intraoperative image, wherein the intraoperative image is composed of part of the preoperative image.

[0044] In one embodiment, if Figure 3 As shown in FIG, the intraoperative image may be a local image of the chest taken during thoracic surgery. Figure 4 As shown, the preoperative image may be a global image of the chest taken before thoracic surgery. Therefore, the intraoperative image may be a medical image containing a local rib region, and the preoperative image may be a medical image containing a complete rib region, that is, the intraoperative image is composed of part of the preoperative image.

[0045] For example, the local image and the global image may be a computed tomography (CT) image, a digital radiography (DR) image, or a magnetic resonance imaging (MRI) image, and the embodiments of the present application do not specifically limit this.

[0046] However, it should be noted that the embodiments of the present application do not specifically limit which tissue or organ of the human body the preoperative images and intraoperative images are images of. They may be images of the chest cavity mentioned above, or images of the brain or legs, etc.

[0047] S220: Based on the puncture needle in the coordinate system of the navigation device, generate a simulated puncture needle in the coordinate system of the preoperative image, wherein the coordinate system of the preoperative image is the same as the coordinate system of the intraoperative image.

[0048] In one embodiment, the navigation device refers to an electromagnetic navigation device, such as an NDI standard device, which can locate an object in real time within the magnetic field at the millimeter level. The coordinate system of the navigation device refers to the coordinate system of the navigation device itself. The coordinate system of the preoperative image refers to the coordinate system of the image itself.

[0049] In one embodiment, since both the preoperative image and the intraoperative image are images formed by imaging human tissues and organs using the same device (eg, CT scanner, etc.), the coordinate system of the preoperative image is the same as the coordinate system of the intraoperative image.

[0050] In one embodiment, after the preoperative image and the navigation device are linked, a simulated puncture needle can be generated in the coordinate system of the preoperative image based on the puncture needle in the coordinate system of the navigation device. That is, the puncture needle in the coordinate system of the navigation device is the puncture needle operated by the medical staff, and the simulated puncture needle in the coordinate system of the preoperative image is the puncture needle in the preoperative image that corresponds to the position of the puncture needle operated by the medical staff.

[0051] However, it should be noted that the embodiments of the present application do not specifically limit how to link the preoperative images and the navigation device, and those skilled in the art may make different choices based on actual needs.

[0052] S230: Sending the simulated puncture needle in the coordinate system of the intraoperative image to the front-end device, so as to locate the position of the simulated puncture needle at the front-end device.

[0053] In one embodiment, since the coordinate system of the preoperative image is the same as the coordinate system of the intraoperative image, obtaining the simulated puncture needle in the coordinate system of the preoperative image through step S220 is equivalent to obtaining the simulated puncture needle in the coordinate system of the intraoperative image, and the position of the simulated puncture needle in the preoperative image and the intraoperative image is the same.

[0054] Therefore, sending a simulated puncture needle in the coordinate system of the intraoperative image to the front-end device enables the front-end device to locate the simulated puncture needle, thereby facilitating medical staff to compare it with the puncture needle in the navigation device to determine the target lesion and puncture path.

[0055] It can be seen that the embodiment of the present application does not need to take multiple intraoperative images of the patient during the operation. The target area can be accurately located through the preoperative images, which not only simplifies the surgical process but also reduces the radiation to the patient.

[0056] In another embodiment of the present application, the method further includes: receiving the intraoperative image sent by an image archiving and communication system.

[0057] The embodiment of the present application adopts a picture archiving and communication system (PACS), which is a system used in hospital imaging departments. The PACS system can store a large number of medical images (such as CT images) generated daily in a digital manner through a variety of interfaces (such as DICOM), thereby realizing automatic monitoring of newly captured images.

[0058] In one embodiment, the intraoperative images can be obtained through a PACS system. Of course, the intraoperative images can also be obtained through a direct CT connection, and the embodiment of the present application does not specifically limit the method for obtaining the intraoperative images. In addition, the acquired intraoperative images can be general medical image format data, such as images in DICOM data format, to facilitate subsequent processing and storage of the intraoperative images.

[0059] In another embodiment, the extracted intraoperative image (i.e., a sequence of CT images) can also be directly sent to a data storage center using a POST request of the Hypertext Transfer Protocol (HTTP). The data storage center is used to store the intraoperative image and also to send the intraoperative image and a simulated puncture needle to a front-end device so as to present the simulated puncture needle together with the intraoperative image to medical staff for viewing.

[0060] In another embodiment of the present application, Figure 5 As shown, Figure 2 The illustrated step S210 includes the following contents.

[0061] S510: Analyze the intraoperative image to obtain patient information of the intraoperative image.

[0062] In one embodiment, intraoperative images are analyzed to obtain patient information included in the intraoperative images, such as patient name, patient number, patient gender, and date of birth, etc. The embodiment of the present application does not specifically limit the patient information.

[0063] S520: Send the patient information to the image archiving and communication system to query whether the preoperative image exists.

[0064] In one embodiment, since the preoperative image and the intraoperative image are images of the same patient, the patient information of the preoperative image and the intraoperative image is the same. The parsed patient information is sent to the PACS system, and it can be queried whether there is a preoperative image with the same patient information as the intraoperative image.

[0065] S530: When the preoperative image exists, extract the preoperative image in the image archiving and communication system.

[0066] In one embodiment, the patient information is sent to the PACS system to query whether there are relevant images, that is, preoperative images. If there are preoperative images, the preoperative images are extracted from the PACS system. If there are no preoperative images, they are not extracted.

[0067] In one embodiment, in order to improve the calculation speed and reduce the calculation time, the preoperative image acquired in the embodiment of the present application may also be a sequence of an image, such as a sequence of a CT image. It should be understood that the sequence is an optimal sequence for characterizing the preoperative image.

[0068] In another embodiment, the extracted preoperative images (i.e., a sequence of CT images) can also be directly sent to a data storage center using a POST request of the Hypertext Transfer Protocol (HTTP). The data storage center is used to store the preoperative images and also to send the preoperative images to the prediction module to predict the segmentation, detection, and registration results of the preoperative images.

[0069] It can be seen that the embodiment of the present application obtains preoperative images and intraoperative images through the PACS system, so that the puncture positioning method provided in the embodiment of the present application is automatically connected to the PACS system, which is convenient for data transmission. At the same time, it can assist the doctor to compare the simulated puncture needle in the intraoperative image with the puncture needle in the navigation device, so as to locate the puncture point.

[0070] In another embodiment of the present application, Figure 6 As shown, Figure 5 The illustrated step S510 includes the following contents.

[0071] S610: Performing data screening on the intraoperative images according to preset configuration rules.

[0072] In one embodiment, before the intraoperative images are subjected to patient information recognition, the acquired intraoperative images may be subjected to data screening or filtering according to preset configuration rules to obtain screened intraoperative images that meet the requirements.

[0073] In one embodiment, the configuration rule may be in the form of a configuration file. The configuration rule may include format regulations (i.e., the influence of the DICOM format). For example, when the intraoperative image format is nii, the data format of the intraoperative image is converted to the DICOM format. The configuration rule may also include image size regulations. For example, when the size of the intraoperative image exceeds a preset size, the intraoperative image may be cropped. It should be noted that the embodiment of the present application does not specifically limit the configuration rules, and those skilled in the art may make different choices based on actual needs.

[0074] S620: Perform patient information recognition on the screened intraoperative images to obtain the patient information of the intraoperative images.

[0075] In one embodiment, after the intraoperative images have completed data screening, screened intraoperative images that meet the requirements (eg, intraoperative images in DICOM format and of appropriate size) are obtained, and patient information can be identified on the screened intraoperative images.

[0076] In one embodiment, the method for identifying patient information may be optical character recognition (OCR), and the present embodiment of the application does not specifically limit the method for identifying patient information. The OCR recognition method can accurately identify the patient information displayed in the intraoperative image, and the patient information may include the patient's name, patient number, patient gender, date of birth, etc.

[0077] It can be seen that obtaining patient information of intraoperative images provides support for obtaining preoperative images in the PACS system.

[0078] In another embodiment of the present application, the method further includes: obtaining segmentation results of tissues and organs in the intraoperative images through a network model based on the intraoperative images; performing three-dimensional reconstruction on the segmentation results to obtain three-dimensional modeling results of the tissues and organs; and sending the three-dimensional modeling results to the front-end device to locate the position of the simulated puncture needle in the three-dimensional modeling results.

[0079] The embodiments of the present application do not specifically limit the type and number of network models. Different network models can be used for different tissues and organs in intraoperative images to obtain tissue and organ segmentation results, and the same network model can be used for different tissues and organs in intraoperative images to obtain tissue and organ segmentation results.

[0080] In one embodiment, a traditional network model based on a traditional threshold method or a watershed method, or a network model based on deep learning can be used to obtain the segmentation result of the lungs.

[0081] In one embodiment, the image corresponding to the lung segmentation result is cut into small blocks, and then segmented using ResUnet to obtain the bronchial segmentation result corresponding to each small block. Finally, the bronchial segmentation results corresponding to all small blocks are connected using the regional growing method to form a complete bronchial segmentation result.

[0082] In addition, the complete bronchial segmentation results can be used to divide the bronchi into 18 segments and 42 subsegments in combination with the topological structure of the bronchi.

[0083] In one embodiment, the image corresponding to the segmentation result of the lung is cut into small blocks, and ResUnet is used, and all blocks therein adopt a bottleneck structure. During the training process, the length of the input data and the output result is increased, and an LSTM module is added to smooth the interlobar fissures, so as to obtain the lung fissure segmentation result corresponding to each small block. Then, the lung fissure segmentation results corresponding to all small blocks are connected by using the method of regional growing to form three complete lung fissure segmentation results.

[0084] In addition, the lung segmentation results and the lung fissure segmentation results are combined to segment the lung into 5 lobes. At the same time, the results of the 18 bronchial segments are used to segment the lung into 18 segments in combination with the characteristics of the watershed.

[0085] In one embodiment, the segmented bronchus is expanded, the loss of the bronchus is transmitted at the first layer of bronchus, and semi-supervised learning is performed at the junction of the second layer of bronchus and the vascular mask. The VAE solution is adopted to ensure the integrity of the structure. The input to the model is the arterial result and the venous result (there are errors). The output of the model includes three branches: an abnormality detection branch, an arterial modification branch, and a venous modification branch. The pulmonary vascular segmentation result is output through the model.

[0086] In one embodiment, a traditional network model based on a traditional threshold method or a watershed method, or a network model based on deep learning may be used to obtain the segmentation result of the epidermis.

[0087] In one embodiment, the UNet model is used to segment the bones, and then the segmented bones are classified into 6 types of bones (ribs / vertebrae / scapula / clavicle / thoracic vertebrae / pelvis), and the ribs are divided into 24 types of ribs.

[0088] Based on the above, the segmentation results of tissues and organs in the intraoperative images were obtained, and the segmentation results of all tissues and organs were reconstructed in three dimensions using the MarchingCube algorithm to obtain the three-dimensional modeling results of the tissues and organs. The three-dimensional modeling results were sent to the front-end device, so that the position of the simulated puncture needle could be located in the three-dimensional modeling results.

[0089] In another embodiment of the present application, Figure 7 As shown, Figure 2 The step S220 shown includes the following contents.

[0090] S710: Convert the coordinate system of the navigation device with the coordinate system of the marker on the preoperative image to obtain a first conversion matrix.

[0091] In one embodiment, the marker is a positioning piece, and the positioning piece can be connected to the navigation device. The marker is arranged on a human body model, and the human body model and the navigation device are arranged in a space.

[0092] In one embodiment, by imaging the tissues and organs of the human body, a preoperative image with markers can be formed. The coordinate system of the markers on the preoperative image refers to the coordinate system where the markers on the image are located, and the coordinate system of the markers on the preoperative image is the same as the coordinate system of the markers on the human body model.

[0093] In one embodiment, when the positioning film is in the magnetic field of the navigation device, the navigation device can return the midpoint of the positioning film and the quaternion describing the posture of the positioning film. The coordinate system of the navigation device and the coordinate system of the marker on the preoperative image are converted through the midpoint and quaternion of the positioning film to obtain a first transformation matrix.

[0094] S720: Convert the coordinate system of the marker on the preoperative image with the coordinate system of the preoperative image to obtain a second conversion matrix.

[0095] In one embodiment, the key points of the marker on the preoperative image are first translated and transformed with the corresponding key points of the standard segmentation to merge the two key points together, and then the X, Y, and Z axes in the coordinate system of the marker on the preoperative image and the X, Y, and Z axes in the coordinate system of the standard segmentation are rotated and transformed using the registration model to determine the rotation angles of the X, Y, and Z axes in the coordinate system of the marker on the preoperative image relative to the X, Y, and Z axes in the coordinate system of the preoperative image, thereby obtaining a second transformation matrix.

[0096] The registration model can be trained by a spatial transformer network (STN). The spatial transformer network can perform spatial transformation on the segmentation result of the marker on the preoperative image based on the standard segmentation, and output a new image, thereby realizing the registration of the segmentation result of the marker and the standard segmentation.

[0097] In one embodiment, the coordinate system of the preoperative image is the same as the coordinate system of the standard segmentation, that is, the directions of the X, Y and Z axes in the coordinate system of the preoperative image are the same as those in the coordinate system of the standard segmentation.

[0098] S730: transforming the puncture needle into the coordinate system of the preoperative image according to the first transformation matrix and the second transformation matrix, so as to generate the simulated puncture needle on the preoperative image.

[0099] like Figure 8 As shown, the navigation coordinate system and the marker coordinate system can be transformed by using the first transformation matrix, and the marker coordinate system and the preoperative image coordinate system can be transformed by using the second transformation matrix. Therefore, in order to make the three-dimensional modeling results of tissues and organs and the navigation device produce a linkage effect, the navigation device coordinate system and the preoperative image coordinate system can be associated, and then this intermediate medium can be used to achieve it, that is, the marker coordinate system in the navigation device.

[0100] In one embodiment, the position of the puncture needle in the coordinate system of the marker can be determined based on the first transformation matrix, and the puncture needle can be transformed to the coordinate system of the preoperative image based on the second transformation matrix and the position of the puncture needle in the coordinate system of the marker to generate a simulated puncture needle on the preoperative image.

[0101] It can be seen that, through the registration model, the three-dimensional modeling results of the tissues and organs after the three-dimensional modeling of the intraoperative images and the navigation device where the puncture needle is located can be accurately and quickly registered.

[0102] In another embodiment of the present application, Figure 7 Step S720 in the method shown includes: performing a translation transformation on the key points of the marker on the preoperative image and the corresponding key points of the standard segmentation; obtaining a rotation transformation matrix between the segmentation result of the marker and the standard segmentation according to the registration model; determining the corresponding coordinate values ​​of the preset points in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image according to the rotation transformation matrix; determining the directions of the X, Y and Z axes of the coordinate system of the preoperative image relative to the X, Y and Z axes of the coordinate system of the marker on the preoperative image according to the corresponding coordinate values ​​of the preset points in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image; and determining the second transformation matrix according to the directions.

[0103] The spatial transformation networks in the prior art all complete three-dimensional affine transformations, namely, translation transformation, scaling transformation and rotation transformation. However, in the present application, in order to improve the registration efficiency of the registration model, the above-mentioned translation transformation and scaling transformation are completed before the segmentation results of the marker are input into the registration model. Therefore, the registration model can only learn the rotation transformation, that is, the output of the positioning network in the spatial transformation network is only the rotation angle.

[0104] In one embodiment, the coordinate value of the preset point in the coordinate system of the preoperative image is multiplied by the rotation transformation matrix to obtain the coordinate value of the preset point in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image.

[0105] In one embodiment, the coordinate value corresponding to the preset point in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image is the new coordinate value of the preset point in the coordinate system of the preoperative image, that is, the preset point in the coordinate system of the preoperative image is transformed into a new coordinate system, that is, the coordinate system of the marker on the preoperative image, through rotation transformation.

[0106] However, it should be noted that the embodiment of the present application does not specifically limit the type and number of preset points, and the preset points may include the midpoint, edge point and / or axis point of the marker.

[0107] In one embodiment, the midpoint and the edge point define the direction of the X axis, and the midpoint and the axis point define the direction of the Y axis, and the Z axis can be obtained by multiplying the Y axis by the X axis. Therefore, according to the new coordinate values ​​of the midpoint, the edge point and / or the axis point, the directions of the X, Y and Z axes of the coordinate system of the preoperative image relative to the X, Y and Z axes of the coordinate system of the marker on the preoperative image can be obtained.

[0108] In one embodiment, the second transformation matrix is ​​composed of the directions of the X, Y, and Z axes of the coordinate system of the preoperative image relative to the X, Y, and Z axes of the coordinate system of the marker on the preoperative image. Alternatively, the second transformation matrix is ​​composed of the directions of the X, Y, and Z axes of the coordinate system of the preoperative image relative to the X, Y, and Z axes of the coordinate system of the marker on the preoperative image and the midpoint of the marker on the preoperative image. This embodiment of the application does not specifically limit this.

[0109] In another embodiment of the present application, the registration model includes a plurality of cascaded spatial transformation networks, the rotation transformation matrix includes a plurality of rotation transformation matrices, one spatial transformation network corresponds to one rotation transformation matrix, and according to the rotation transformation matrix, the coordinate value corresponding to the preset point in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image is determined, including: calculating the intersection and union ratio of the input and output of each of the plurality of spatial transformation networks; multiplying the rotation transformation matrix corresponding to the spatial transformation network whose intersection and union ratio is less than a preset threshold by the coordinate value of the preset point in the coordinate system of the preoperative image to obtain the coordinate value corresponding to the preset point in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image.

[0110] In one embodiment, if the number of the plurality of spatial transformation networks is N, one spatial transformation network corresponds to one input and one output, and each time a spatial transformation network is passed, a rotation transformation matrix may be generated.

[0111] In one embodiment, N results may be obtained by calculating the intersection-and-union ratios of inputs and outputs of N spatial transformation networks.

[0112] In one embodiment, N results are compared with a preset threshold in order. When the intersection and union ratio corresponding to a certain spatial transformation network (for example, the 10th spatial transformation network) is greater than or equal to the preset threshold, the comparison is stopped, and the rotation transformation matrices (A1 to A10) corresponding to the 1st spatial transformation network to the 10th spatial transformation network are multiplied in sequence with the coordinate value (x, y, z, 1) of the preset point in the coordinate system of the preoperative image, that is, (x, y, z, 1)*A1*A2*…*A10, so as to obtain the coordinate value of the preset point in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image.

[0113] In one embodiment, the N results are compared with preset thresholds respectively to determine which of the N spatial transformation networks have an intersection-and-joint ratio greater than or equal to the preset threshold, and the rotation transformation matrices corresponding to those spatial transformation networks whose intersection-and-joint ratio is greater than or equal to the preset threshold are multiplied in sequence with the coordinate values ​​of the preset points in the coordinate system of the preoperative image, so as to obtain the coordinate values ​​corresponding to the preset points in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image.

[0114] However, it should be noted that the embodiment of the present application does not specifically limit the method for selecting the spatial transformation network whose intersection-over-union ratio is less than the preset threshold, and those skilled in the art may make different selections according to actual needs.

[0115] Exemplary Devices

[0116] The device embodiments of the present application can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0117] Fig. 9 FIG. 1 is a block diagram of a puncture positioning device provided by an embodiment of the present application. Fig. 9 As shown, the device 900 includes:

[0118] An acquisition module 910 is configured to acquire a preoperative image based on the intraoperative image, wherein the intraoperative image is composed of part of the preoperative image;

[0119] A generating module 920, configured to generate a simulated puncture needle in the coordinate system of the preoperative image based on the puncture needle in the coordinate system of the navigation device, wherein the coordinate system of the preoperative image is the same as the coordinate system of the intraoperative image;

[0120] The first sending module 930 is configured to send the simulated puncture needle located in the coordinate system of the intraoperative image to the front-end device, so as to locate the position of the simulated puncture needle at the front-end device.

[0121] In another embodiment of the present application, the device further includes: an information pulling module configured to receive the intraoperative image sent by the image archiving and communication system.

[0122] In another embodiment of the present application, the acquisition module 910 is further configured to: analyze the intraoperative image to obtain the patient information of the intraoperative image; send the patient information to the image archiving and communication system to query whether the preoperative image exists; when the preoperative image exists, extract the preoperative image from the image archiving and communication system.

[0123] In another embodiment of the present application, when the acquisition module 910 analyzes the intraoperative image to obtain the patient information of the intraoperative image, it is further configured to: perform data screening on the intraoperative image according to preset configuration rules; and perform patient information identification on the screened intraoperative image to obtain the patient information of the intraoperative image.

[0124] In another embodiment of the present application, the patient information includes patient name, patient number, patient gender and / or date of birth.

[0125] In another embodiment of the present application, the device also includes: a segmentation module, configured to obtain the segmentation result of the tissue and organ in the intraoperative image through a network model based on the intraoperative image; a reconstruction module, configured to perform three-dimensional reconstruction on the segmentation result to obtain the three-dimensional modeling result of the tissue and organ; a second sending module, configured to send the three-dimensional modeling result to the front-end device to locate the position of the simulated puncture needle in the three-dimensional modeling result.

[0126] In another embodiment of the present application, the generation module 920 is further configured to: transform the coordinate system of the navigation device with the coordinate system of the marker on the preoperative image to obtain a first transformation matrix; transform the coordinate system of the marker on the preoperative image with the coordinate system of the preoperative image to obtain a second transformation matrix; and transform the puncture needle into the coordinate system of the preoperative image according to the first transformation matrix and the second transformation matrix to generate the simulated puncture needle on the preoperative image.

[0127] In another embodiment of the present application, when the generation module 920 transforms the coordinate system of the marker on the preoperative image with the coordinate system of the preoperative image to obtain a second transformation matrix, the generation module 920 is further configured to: perform a translation transformation on the key points of the marker on the preoperative image and the corresponding key points of the standard segmentation; obtain a rotation transformation matrix between the segmentation result of the marker and the standard segmentation according to the registration model; determine the coordinate values ​​corresponding to the preset points in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image according to the rotation transformation matrix; determine the directions of the X, Y and Z axes of the coordinate system of the preoperative image relative to the X, Y and Z axes of the coordinate system of the marker on the preoperative image according to the coordinate values ​​corresponding to the preset points in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image; and determine the second transformation matrix according to the directions.

[0128] In another embodiment of the present application, the registration model includes a plurality of cascaded spatial transformation networks, the rotation transformation matrix includes a plurality of rotation transformation matrices, one spatial change network corresponds to one rotation transformation matrix, and the generation module 920 is further configured to: calculate the intersection and union ratio of the input and output of each of the plurality of spatial transformation networks; multiply the rotation transformation matrix corresponding to the spatial transformation network whose intersection and union ratio is less than a preset threshold by the coordinate value of the preset point in the coordinate system of the preoperative image to obtain the coordinate value of the preset point in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image.

[0129] In another embodiment of the present application, the preset point includes the midpoint, edge point and / or axis point of the marker.

[0130] Exemplary Electronic Devices

[0131] Below, reference Fig.10 To describe an electronic device according to an embodiment of the present application. Fig.10 A block diagram of an electronic device according to an embodiment of the present application is illustrated.

[0132] like Fig.10 As shown, electronic device 1000 includes one or more processors 1010 and memory 1020 .

[0133] The processor 1010 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1000 to perform desired functions.

[0134] The memory 1020 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1010 may run the program instructions to implement the puncture positioning method of each embodiment of the present application described above and / or other desired functions. Various contents such as a first conversion matrix and a second conversion matrix may also be stored in the computer-readable storage medium.

[0135] In one example, the electronic device 1000 may further include: an input device 1030 and an output device 1040 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0136] For example, the input device 1030 may be the microphone or microphone array mentioned above, which is used to capture the input signal of the sound source. When the electronic device is a stand-alone device, the input device 1030 may be a communication network connector.

[0137] In addition, the input device 1030 may also include, for example, a keyboard, a mouse, and the like.

[0138] The output device 1040 can output various information to the outside, including the determined rotation transformation matrix, etc. The output device 1040 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0139] Of course, to simplify, Fig.10 Only some of the components related to the present application in the electronic device 1000 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application conditions, the electronic device 1000 may also include any other appropriate components.

[0140] Exemplary computer program products and computer-readable storage media

[0141] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the puncture positioning method according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0142] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0143] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the puncture positioning method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0144] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0145] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A puncture positioning method, characterized in that: include: Acquire a preoperative image based on the intraoperative image, wherein the intraoperative image is a partial image of human tissues and organs contained in the preoperative image taken during the operation; According to a first conversion matrix and a second conversion matrix, the puncture needle in the coordinate system based on the navigation device is transformed into the coordinate system of the preoperative image to generate a simulated puncture needle on the preoperative image, wherein the first conversion matrix is ​​used to convert the coordinate system of the navigation device and the coordinate system of the marker on the preoperative image; the second conversion matrix is ​​used to convert the coordinate system of the marker on the preoperative image and the coordinate system of the preoperative image; the coordinate system of the preoperative image is the same as the coordinate system of the intraoperative image; The second transformation matrix is ​​determined based on the registration model and the rotation transformation matrix between the segmentation result of the marker obtained based on the registration model and the standard segmentation; wherein the registration model includes a plurality of cascaded spatial transformation networks; the rotation transformation matrix is ​​used to determine the coordinate value corresponding to the preset point in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image; the rotation transformation matrix includes a plurality of rotation transformation matrices, and one spatial variation network corresponds to one rotation transformation matrix; Determining the coordinate value of the preset point in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image according to the rotation transformation matrix, including: calculating the intersection-and-union ratio of the input and output of each of the multiple spatial transformation networks; multiplying the rotation transformation matrix corresponding to the spatial transformation network whose intersection-and-union ratio is less than a preset threshold by the coordinate value of the preset point in the coordinate system of the preoperative image, so as to obtain the coordinate value of the preset point in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image; The three-dimensional modeling results of the simulated puncture needle and the tissue organs in the intraoperative image in the coordinate system of the intraoperative image are sent to the front-end device, so that the front-end device can locate the position of the simulated puncture needle in the three-dimensional modeling results and determine the target lesion and puncture path.

2. The method according to claim 1, characterized in that Also includes: receiving the intraoperative image sent by an image archiving and communication system, Wherein, obtaining preoperative images based on intraoperative images includes: Analyzing the intraoperative images to obtain patient information of the intraoperative images; Sending the patient information to the image archiving and communication system to query whether the preoperative image exists; When the preoperative image exists, the preoperative image is retrieved in the image archiving and communication system.

3. The method according to claim 2, characterized in that The step of analyzing the intraoperative image to obtain patient information of the intraoperative image includes: According to preset configuration rules, data screening is performed on the intraoperative images; Patient information is identified on the screened intraoperative images to obtain patient information of the intraoperative images.

4. The method according to claim 2, characterized in that: The patient information includes patient name, patient number, patient gender and / or date of birth.

5. The method according to claim 1, characterized in that Also includes: According to the intraoperative image, obtaining a segmentation result of tissues and organs in the intraoperative image through a network model; The segmentation result is three-dimensionally reconstructed to obtain a three-dimensional modeling result of the tissue organ.

6. The method according to any one of claims 1 to 5, characterized in that The puncture needle in the coordinate system of the navigation device is used to generate a simulated puncture needle in the coordinate system of the preoperative image, including: Transforming the coordinate system of the navigation device with the coordinate system of the marker on the preoperative image to obtain a first transformation matrix; The coordinate system of the marker on the preoperative image is transformed with the coordinate system of the preoperative image to obtain a second transformation matrix.

7. The method according to claim 6, characterized in that The step of converting the coordinate system of the marker on the preoperative image with the coordinate system of the preoperative image to obtain a second conversion matrix includes: Performing a translation transformation between the key points of the marker on the preoperative image and the corresponding key points of the standard segmentation; According to the registration model, obtaining a rotation transformation matrix between the segmentation result of the marker and the standard segmentation; Determine, according to the rotation transformation matrix, a coordinate value corresponding to a preset point in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image; Determine the directions of the X, Y and Z axes of the coordinate system of the preoperative image relative to the X, Y and Z axes of the coordinate system of the marker on the preoperative image according to the coordinate values ​​corresponding to the preset points in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image; The second transformation matrix is ​​determined according to the direction.

8. The method according to claim 7, characterized in that The preset points include the midpoint, edge point and / or axis point of the marker.

9. A puncture positioning device, characterized in that: include: An acquisition module, configured to acquire a preoperative image based on the intraoperative image, wherein the intraoperative image is a partial image of human tissues and organs contained in the preoperative image taken during the operation; A generating module, configured to transform the puncture needle in the coordinate system based on the navigation device into the coordinate system of the preoperative image according to a first conversion matrix and a second conversion matrix, so as to generate a simulated puncture needle on the preoperative image, wherein the first conversion matrix is ​​used to convert the coordinate system of the navigation device and the coordinate system of the marker on the preoperative image; the second conversion matrix is ​​used to convert the coordinate system of the marker on the preoperative image and the coordinate system of the preoperative image; the coordinate system of the preoperative image is the same as the coordinate system of the intraoperative image; The second transformation matrix is ​​determined based on the registration model and the rotation transformation matrix between the segmentation result of the marker obtained based on the registration model and the standard segmentation; wherein the registration model includes a plurality of cascaded spatial transformation networks; the rotation transformation matrix is ​​used to determine the coordinate value corresponding to the preset point in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image; the rotation transformation matrix includes a plurality of rotation transformation matrices, and one spatial variation network corresponds to one rotation transformation matrix; Determining the coordinate value of the preset point in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image according to the rotation transformation matrix, including: calculating the intersection-and-union ratio of the input and output of each of the multiple spatial transformation networks; multiplying the rotation transformation matrix corresponding to the spatial transformation network whose intersection-and-union ratio is less than a preset threshold by the coordinate value of the preset point in the coordinate system of the preoperative image, so as to obtain the coordinate value of the preset point in the coordinate system of the preoperative image in the coordinate system of the marker on the preoperative image; The first sending module is configured to send the simulated puncture needle located in the coordinate system of the intraoperative image to the front-end device, so as to locate the position of the simulated puncture needle in the front-end device and determine the target lesion and the puncture path.

10. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having computer program instructions stored thereon, wherein when the computer program instructions are executed by a processor, the processor is enabled to perform the method according to any one of claims 1 to 8.

Images