Probe position prediction and model building method, system, device and medium

Abstract

The present disclosure provides a method, system, device and medium for predicting and building a probe position model. The construction method includes establishing a three-dimensional body part model corresponding to the sample body part based on sample detection data corresponding to the sample body part, simulating virtual probe positioning data and virtual probe scanning data corresponding to the virtual probe according to the three-dimensional data corresponding to the three-dimensional body part model, obtaining a two-dimensional structure sample data set corresponding to the sample body part based on the virtual probe positioning data and the virtual probe scanning data, and training a probe position prediction model based on the two-dimensional structure sample data set. The present disclosure predicts the actual detection probe corresponding to the input actual detection data through the probe position prediction model, and the actual positioning data when detecting the actual body part, so as to realize accurate and reliable positioning of the actual detection probe.

Description

Technical Field

[0001] The present disclosure relates to the field of ultrasound technology, and in particular to a probe position prediction and model building method, system, device and medium. Background Art

[0002] In the medical field, when examining a body part of a patient, the patient is usually examined with the aid of a detection probe.

[0003] With the continuous advancement of technology, ultrasound technology has been widely used in the medical field and has many representative application fields. These include type A ultrasound detection, type B ultrasound detection, and intravascular ultrasound detection. Type A ultrasound detection can be used to evaluate cardiac and hemodynamic status, while type B ultrasound detection is used for biological tissue imaging, such as gynecology and fetal monitoring. In addition, intravascular ultrasound detection is also widely used in catheter interventional surgery to help doctors observe conditions such as vascular stenosis or thrombosis. In addition to the medical field, ultrasound technology also plays an important role in many other fields, such as materials science, mechanical engineering, and food safety. Due to its non-invasive, real-time and repeatable characteristics, ultrasound technology will continue to show broad application prospects in various fields.

[0004] Existing detection probes, especially ultrasound probes, have large probe positioning errors when examining or treating a patient's body parts, and cannot achieve accurate and reliable positioning, which affects the treatment effect. Summary of the invention

[0005] The technical problem to be solved by the present disclosure is to overcome the defect in the prior art that the detection probe cannot be positioned in real time and accurately, and to provide a probe position prediction and model building method, system, device and medium.

[0006] The present invention solves the above technical problems through the following technical solutions:

[0007] In a first aspect, a method for constructing a probe position prediction model is provided, the method comprising:

[0008] Based on the sample detection data corresponding to the sample body part, establishing a three-dimensional body part model corresponding to the sample body part;

[0009] Simulating virtual probe positioning data and virtual probe scanning data corresponding to the virtual probe according to the three-dimensional data corresponding to the three-dimensional body part model;

[0010] Based on the virtual probe positioning data and the virtual probe scanning data, obtaining a two-dimensional structure sample data set corresponding to the sample body part;

[0011] The probe position prediction model is obtained by training according to the two-dimensional structure sample data set;

[0012] The probe position prediction model is used to predict the actual positioning data of the actual detection probe corresponding to the input actual detection data when detecting the actual body part;

[0013] The actual body part includes a first body part that the actual detection probe can detect, or a second body part that the actual detection probe cannot detect.

[0014] Preferably, the step of simulating virtual probe positioning data corresponding to the virtual detection probe and virtual probe scanning data according to the three-dimensional data corresponding to the three-dimensional body part model comprises:

[0015] Simulate the virtual probe positioning data corresponding to the virtual probe detecting the first body part and the virtual probe scanning data according to the three-dimensional data corresponding to the three-dimensional body part model;

[0016] The probe position prediction model is used to predict the actual positioning data of the actual detection probe corresponding to the input actual detection data, which is probed into the first body part;

[0017] or, simulating the virtual probe positioning data and the virtual probe scanning data corresponding to the virtual probe for detecting the second body part according to the three-dimensional data corresponding to the three-dimensional body part model;

[0018] The probe position prediction model is used to predict the actual positioning data of the actual detection probe corresponding to the input actual detection data, which is inserted into the body and outside the second body part.

[0019] Preferably, the step of establishing a three-dimensional body part model corresponding to the sample body part based on the sample detection data corresponding to the sample body part comprises:

[0020] Scanning the target object based on the sample detection probe to collect sample detection data of the target object;

[0021] The sample detection data is subjected to three-dimensional reconstruction processing to establish the three-dimensional body part model.

[0022] Preferably, the step of simulating virtual probe positioning data corresponding to the virtual probe and virtual probe scanning data according to the three-dimensional data corresponding to the three-dimensional body part model comprises:

[0023] Get preset scan data;

[0024] Simulating the virtual probe positioning data corresponding to the virtual probe;

[0025] The virtual probe scanning data is obtained based on the virtual probe positioning data and the preset scanning data.

[0026] Preferably, when the first body part includes a heart, the step of obtaining preset scan data further includes:

[0027] Acquiring a preset position range of the virtual probe in the heart area;

[0028] Wherein, the virtual probe positioning data is located within the preset position range.

[0029] Preferably, the step of acquiring the two-dimensional structure sample data set corresponding to the sample body part based on the virtual probe positioning data and the virtual probe scanning data comprises:

[0030] For any preset sampling point of the virtual probe in the three-dimensional body part model, based on the intersection of the virtual probe scanning data corresponding to the preset sampling point and the three-dimensional body part model, obtaining initial three-dimensional intersection data corresponding to the preset sampling point;

[0031] Based on the virtual probe positioning data corresponding to the preset sampling point and the initial three-dimensional intersection data, obtaining the initial two-dimensional structure data corresponding to the preset sampling point;

[0032] Obtaining a random transformation matrix corresponding to the preset sampling point;

[0033] The two-dimensional structure sample data set is obtained based on a plurality of the random transformation matrices and the corresponding initial two-dimensional structure data.

[0034] Preferably, the step of obtaining the random transformation matrix corresponding to the preset sampling point includes:

[0035] Obtaining a random rotation matrix corresponding to the preset sampling point;

[0036] Obtaining the target position coordinates corresponding to the preset sampling point;

[0037] The random transformation matrix is ​​obtained based on the random rotation matrix and the target position coordinates.

[0038] Preferably, the first body part includes the heart or cardiovascular system; the second body part includes the liver, kidney, thyroid gland, breast or prostate gland;

[0039] And / or, the detection probe includes an ultrasonic probe.

[0040] Preferably, the step of training the probe position prediction model according to the two-dimensional structure sample data set includes:

[0041] A preset network model is trained based on the two-dimensional structure sample data set to obtain the probe position prediction model.

[0042] Preferably, the preset network model is constructed based on an Encoder (encoding) structure model and a Decoder (decoding) structure model.

[0043] Preferably, the preset network model is constructed based on UNet (a segmentation model) network and PointNet++ (a deep learning model) network.

[0044] Preferably, the probe position prediction model includes an input layer, a downsampling layer, an upsampling layer and an output layer;

[0045] The step of training a preset network model based on the two-dimensional structure sample data set to obtain the probe position prediction model includes:

[0046] Inputting the two-dimensional structure sample data set into the input layer for sampling operation, and outputting the number of initial actual point sets corresponding to the initial two-dimensional structure data;

[0047] If the number of the initial actual point set is greater than the number of the preset point set, the initial two-dimensional structure data is input to the downsampling layer, and the target two-dimensional structure data is output;

[0048] If the number of the initial actual point set is less than the number of the preset point set, inputting the initial two-dimensional structure data to the upsampling layer, and outputting the target two-dimensional structure data;

[0049] The number of actual target point sets corresponding to the target two-dimensional structure data is equal to the number of preset point sets;

[0050] The target two-dimensional structure data is input into the output layer, and the actual positioning data when the actual detection probe detects the actual body part is output.

[0051] Preferably, the step of inputting the initial two-dimensional structure data into the downsampling layer and outputting the target two-dimensional structure data comprises:

[0052] Using a multi-layer perceptron and a farthest point sampling method, the initial two-dimensional structure data is processed to output the target two-dimensional structure data;

[0053] Wherein, the target two-dimensional structure data includes layer feature information and image position information;

[0054] and / or,

[0055] The step of inputting the initial two-dimensional structure data into the upsampling layer and outputting the target two-dimensional structure data comprises:

[0056] Using a multi-layer perceptron and contour interpolation method to process the initial two-dimensional structure data, and output the target two-dimensional structure data;

[0057] The target two-dimensional structure data includes layer feature information and image position information.

[0058] In a second aspect, a probe position prediction method is also provided, the probe position prediction method comprising the method for constructing the probe position prediction model described above, the probe position prediction method comprising:

[0059] Based on the initial detection data collected by the actual detection probe, target detection data corresponding to the actual body part is obtained;

[0060] The target detection data is input into the probe position prediction model obtained based on the construction method to output the actual positioning data when the actual detection probe detects the actual body part. Preferably, the step of obtaining the target detection data based on the initial detection data collected by the actual detection probe includes:

[0061] Acquiring the initial detection data collected by the actual detection probe;

[0062] Perform semantic segmentation operations on the initial detection data to obtain several segmentation mask images;

[0063] Performing contour extraction operation on the plurality of segmentation mask images to obtain the target detection data. Preferably, the step of performing semantic segmentation operation on the initial detection data to obtain the plurality of segmentation mask images comprises:

[0064] Performing a semantic segmentation operation on the initial detection data based on a preset semantic segmentation model to obtain a plurality of segmentation mask images;

[0065] and / or,

[0066] The step of performing contour extraction operation on the plurality of segmentation mask images to obtain the target detection data comprises:

[0067] Traversing all foreground pixels in the segmentation mask image to obtain initial edge contour points;

[0068] Traversing the initial edge contour points, and determining whether each target edge contour point in the initial edge contour points is a preset contour point;

[0069] If so, the target edge contour points are removed from the initial edge contour points, and the remaining edge contour points after removing the target edge contour points from the initial edge contour points are used as the target detection data.

[0070] In a third aspect, a system for constructing a probe position prediction model is provided, the system comprising:

[0071] A model building module, used to build a three-dimensional body part model corresponding to the sample body part based on the sample detection data corresponding to the sample body part;

[0072] A virtual probe simulation module, used to simulate virtual probe positioning data and virtual probe scanning data corresponding to the virtual probe according to the three-dimensional data corresponding to the three-dimensional body part model;

[0073] A sample acquisition module, used for acquiring a two-dimensional structure sample data set corresponding to the sample body part based on the virtual probe positioning data and the virtual probe scanning data;

[0074] A model training module, used for training the probe position prediction model according to the two-dimensional structure sample data set;

[0075] The probe position prediction model is used to predict the actual positioning data of the actual detection probe corresponding to the input actual detection data when detecting the actual body part;

[0076] The actual body part includes a first body part that the actual detection probe can detect, or a second body part that the actual detection probe cannot detect. Preferably, the virtual probe simulation module is used to simulate the virtual probe positioning data corresponding to the virtual probe detecting the first body part and the virtual probe scanning data according to the three-dimensional data corresponding to the three-dimensional body part model;

[0077] The probe position prediction model is used to predict the actual positioning data of the actual detection probe corresponding to the input actual detection data, which is probed into the first body part;

[0078] Or, the virtual probe simulation module is used to simulate the virtual probe positioning data and the virtual probe scanning data corresponding to the virtual probe detecting the second body part according to the three-dimensional data corresponding to the three-dimensional body part model;

[0079] The probe position prediction model is used to predict the actual positioning data of the actual detection probe corresponding to the input actual detection data, which is inserted into the body and outside the second body part.

[0080] Preferably, the model building module includes:

[0081] A data acquisition unit, configured to scan a target object based on a sample detection probe to acquire sample detection data of the target object;

[0082] The model building unit is used to perform three-dimensional reconstruction processing on the sample detection data to build the three-dimensional body part model.

[0083] Preferably, the virtual probe simulation module includes:

[0084] A preset data acquisition unit, used for acquiring preset scanning data;

[0085] A positioning data acquisition unit, used for simulating the virtual probe positioning data corresponding to the virtual probe;

[0086] The scanning data acquisition unit is used to obtain the virtual probe scanning data based on the virtual probe positioning data and the preset scanning data.

[0087] Preferably, when the first body part includes a heart, the virtual probe simulation module further includes:

[0088] A position range acquisition unit, used for acquiring a preset position range of the virtual probe in the heart area;

[0089] Wherein, the virtual probe positioning data is located within the preset position range.

[0090] Preferably, the sample acquisition module includes:

[0091] a three-dimensional intersection point acquisition unit, configured to obtain, for any preset sampling point of the virtual probe in the three-dimensional body part model, initial three-dimensional intersection point data corresponding to the preset sampling point based on the intersection of the virtual probe scanning data corresponding to the preset sampling point and the three-dimensional body part model;

[0092] An initial data acquisition unit, configured to obtain initial two-dimensional structure data corresponding to the preset sampling point based on the virtual probe positioning data corresponding to the preset sampling point and the initial three-dimensional intersection data;

[0093] A transformation matrix acquisition unit, used to acquire a random transformation matrix corresponding to the preset sampling point;

[0094] The sample data construction unit is used to obtain the two-dimensional structure sample data set based on a plurality of the random transformation matrices and the corresponding initial two-dimensional structure data.

[0095] Preferably, the transformation matrix acquisition unit is further used to acquire a random rotation matrix corresponding to the preset sampling point; acquire the target position coordinates corresponding to the preset sampling point; and obtain the random transformation matrix based on the random rotation matrix and the target position coordinates.

[0096] Preferably, the first body part includes the heart or cardiovascular system; the second body part includes the liver, kidney, thyroid gland, breast or prostate gland;

[0097] And / or, the detection probe includes an ultrasonic probe.

[0098] Preferably, the model training module is used to train a preset network model based on the two-dimensional structure sample data set to obtain the probe position prediction model.

[0099] Preferably, the preset network model is constructed based on an Encoder structure model and a Decoder structure model.

[0100] Preferably, the preset network model is constructed based on a UNet network and a PointNet++ network.

[0101] Preferably, the probe position prediction model includes an input layer, a downsampling layer, an upsampling layer and an output layer; the model training module is used to input the two-dimensional structure sample data set into the input layer for sampling operation, and output the initial actual point set number corresponding to the initial two-dimensional structure data; if the initial actual point set number is greater than the preset point set number, the initial two-dimensional structure data is input into the downsampling layer, and the target two-dimensional structure data is output; if the initial actual point set number is less than the preset point set number, the initial two-dimensional structure data is input into the upsampling layer, and the target two-dimensional structure data is output; wherein the target actual point set number corresponding to the target two-dimensional structure data is equal to the preset point set number; the target two-dimensional structure data is input into the output layer, and the actual positioning data when the actual detection probe detects the actual body part is output.

[0102] Preferably, the model training module uses a multi-layer perceptron and a farthest point sampling method to process the initial two-dimensional structure data and output the target two-dimensional structure data;

[0103] Wherein, the target two-dimensional structure data includes layer feature information and image position information;

[0104] And / or, the model training module uses a multi-layer perceptron and contour interpolation method to process the initial two-dimensional structure data and output the target two-dimensional structure data;

[0105] The target two-dimensional structure data includes layer feature information and image position information.

[0106] In a fourth aspect, a probe position prediction system is further provided, the probe position prediction system comprising the construction system of the probe position prediction model described above, the probe position prediction system comprising:

[0107] A detection data acquisition module, used to obtain target detection data corresponding to the actual body part based on the initial detection data collected by the actual detection probe;

[0108] A probe position prediction module is used to input the target detection data into a probe position prediction model obtained based on the construction system to output actual positioning data when the actual detection probe detects the actual body part.

[0109] Preferably, the detection data acquisition module includes:

[0110] An initial data acquisition unit, used for acquiring the initial detection data acquired by the actual detection probe;

[0111] A semantic segmentation unit, used for performing a semantic segmentation operation on the initial detection data to obtain a plurality of segmentation mask images;

[0112] The contour extraction unit is used to perform contour extraction operations on the plurality of segmentation mask images to obtain the target detection data.

[0113] Preferably, the semantic segmentation unit is used to perform a semantic segmentation operation on the initial detection data based on a preset semantic segmentation model to obtain a plurality of segmentation mask images;

[0114] And / or, the contour extraction unit is used to traverse all foreground pixel points in the segmentation mask image to obtain initial edge contour points; traverse the initial edge contour points to determine whether each target edge contour point in the initial edge contour points is a preset contour point; if so, remove the target edge contour points from the initial edge contour points, and use the remaining edge contour points after removing the target edge contour points from the initial edge contour points as the target detection data.

[0115] In a fifth aspect, a probe device is also provided, the probe device comprising a detection probe and the above-mentioned probe position prediction system.

[0116] In a sixth aspect, an electronic device is also provided, comprising a memory, a processor, and a computer program stored in the memory and used to run on the processor, wherein when the processor executes the computer program, the method for constructing the probe position prediction model mentioned above, or the method for predicting the probe position mentioned above is implemented.

[0117] In a seventh aspect, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the method for constructing the probe position prediction model or the method for predicting the probe position is implemented.

[0118] On the basis of being in accordance with the common sense in the art, the above-mentioned preferred conditions can be arbitrarily combined to obtain the preferred embodiments of the present disclosure.

[0119] The positive and progressive effects of this disclosure are:

[0120] The probe position prediction and model building method, system, device and medium disclosed in the present invention establish a three-dimensional body part model corresponding to the sample body part based on the sample detection data, simulate the virtual probe positioning data and virtual probe scanning data corresponding to the virtual probe according to the three-dimensional data corresponding to the three-dimensional body part model, and finally obtain the two-dimensional structure sample data set corresponding to the sample body part, train to obtain the probe position prediction model, and use the probe position prediction model to predict the actual detection probe corresponding to the input actual detection data, and the actual positioning data when detecting the actual body part, so as to achieve accurate and reliable positioning of the actual detection probe. BRIEF DESCRIPTION OF THE DRAWINGS

[0121] Figure 1 A first flow chart of a method for constructing a probe position prediction model provided in Embodiment 1 of the present disclosure;

[0122] Figure 2 A second flow chart of the method for constructing a probe position prediction model provided in Embodiment 1 of the present disclosure;

[0123] Figure 3 A third flow chart of the method for constructing a probe position prediction model provided in Embodiment 1 of the present disclosure;

[0124] Figure 4 A fourth flow chart of the method for constructing a probe position prediction model provided in Embodiment 1 of the present disclosure;

[0125] Figure 5 A first schematic diagram of establishing a three-dimensional body part model in the method for constructing a probe position prediction model provided in Embodiment 1 of the present disclosure;

[0126] Figure 6 A second schematic diagram of establishing a three-dimensional body part model in the method for constructing a probe position prediction model provided in Embodiment 1 of the present disclosure;

[0127] Figure 7 A fifth flow chart of the method for constructing a probe position prediction model provided in Embodiment 1 of the present disclosure;

[0128] Figure 8A sixth flow chart of the method for constructing a probe position prediction model provided in Embodiment 1 of the present disclosure;

[0129] Fig. 9 A schematic diagram of an image corresponding to the initial two-dimensional structure data in the method for constructing a probe position prediction model provided in Example 1 of the present disclosure;

[0130] Fig.10 A schematic diagram of a probe position prediction model training process in a method for constructing a probe position prediction model provided in Example 1 of the present disclosure;

[0131] Fig.11 A model structure diagram of a probe position prediction model in the center of the method for constructing a probe position prediction model provided in Example 1 of the present disclosure;

[0132] Fig.12 A schematic diagram of a first flow chart of a probe position prediction method provided in Embodiment 2 of the present disclosure;

[0133] Fig.13 A second flow chart of the probe position prediction method provided in Embodiment 2 of the present disclosure;

[0134] Fig.14 A schematic diagram of the prediction process of the probe position prediction method provided in Embodiment 2 of the present disclosure;

[0135] Fig.15 A schematic diagram of an initial intracardiac two-dimensional ultrasound image in the probe position prediction method provided in Example 2 of the present disclosure;

[0136] Fig.16 A schematic diagram of a segmentation mask image in the probe position prediction method provided in Embodiment 2 of the present disclosure;

[0137] Fig.17 A schematic diagram of a contour map formed by initial edge contour points in the probe position prediction method provided in Example 2 of the present disclosure;

[0138] Fig.18 A schematic diagram of a target intracardiac two-dimensional ultrasound image in the probe position prediction method provided in Example 2 of the present disclosure;

[0139] Fig.19 A schematic diagram of the structure of a system for constructing a probe position prediction model provided in Embodiment 3 of the present disclosure;

[0140] Fig. 20 A schematic diagram of the structure of a probe position prediction system provided in Embodiment 4 of the present disclosure;

[0141] Fig.21 A schematic diagram of the structure of an electronic device provided in Example 6 of the present disclosure. DETAILED DESCRIPTION

[0142] The present disclosure is further described below by way of examples, but the present disclosure is not limited to the scope of the examples.

[0143] Example 1

[0144] In order to perform cardiac ablation surgery, an interventional catheter needs to be placed into the heart through a blood vessel, and the key is to accurately determine the position of the catheter probe. At present, the commonly used catheter positioning methods are three-dimensional magnetic field positioning and three-dimensional electric field positioning. The magnetic field catheter positioning method uses three coils under the operating table to generate a magnetic field, and senses the strength of these magnetic fields through the detection coils near the catheter to determine the position of the catheter in the magnetic field. However, this method requires that the patient cannot move, otherwise it will cause a large positioning error. The electric field positioning method uses three pairs of electrodes attached to the patient's skin to generate three orthogonal electric fields. The electrodes embedded in the catheter head are used to measure the strength of these electric fields to determine the position of the catheter head in the heart. Compared with magnetic field positioning, since the electrodes are located on the surface of the skin, the positioning error is smaller when the patient changes position. Both methods require expensive equipment, which can only be produced by a few manufacturers.

[0145] This embodiment solves the problems existing in the above existing solutions and provides a method for constructing a probe position prediction model. Figure 1 As shown, the construction method includes:

[0146] S1. Based on the sample detection data corresponding to the sample body part, a three-dimensional body part model corresponding to the sample body part is established.

[0147] S2. According to the three-dimensional data corresponding to the three-dimensional body part model, simulate the virtual probe positioning data and the virtual probe scanning data corresponding to the virtual probe.

[0148] S3. Based on the virtual probe positioning data and the virtual probe scanning data, a two-dimensional structure sample data set corresponding to the sample body part is obtained.

[0149] S4. A probe position prediction model is obtained by training according to the two-dimensional structure sample data set.

[0150] Among them, the probe position prediction model is used to predict the actual detection probe corresponding to the input actual detection data, and the actual positioning data when detecting the actual body part; the actual body part includes the first body part that the actual detection probe can detect, or the second body part that the actual detection probe cannot detect.

[0151] The method for constructing the probe position prediction model of the present embodiment establishes a three-dimensional body part model corresponding to the sample body part based on the sample detection data corresponding to the sample body part, simulates the virtual probe positioning data and the virtual probe scanning data corresponding to the virtual probe according to the three-dimensional data corresponding to the three-dimensional body part model, and finally obtains the two-dimensional structure sample data set corresponding to the sample body part, and then trains the probe position prediction model based on the two-dimensional structure sample data set, predicts the actual detection probe corresponding to the input actual detection data through the probe position prediction model, and the actual positioning data when detecting the actual body part, thereby realizing accurate and reliable positioning of the actual detection probe.

[0152] In an optional embodiment, if Figure 2 As shown, step S2 includes:

[0153] S21. According to the three-dimensional data corresponding to the three-dimensional body part model, simulate the virtual probe positioning data and virtual probe scanning data corresponding to the virtual probe for detecting the first body part.

[0154] The probe position prediction model is used to predict actual positioning data of an actual detection probe corresponding to the input actual detection data, which is probed into the first body part.

[0155] Specifically, the first body part includes the heart or cardiovascular system. During the detection process, the actual detection probe can be inserted into the first body part, that is, the actual detection probe can enter the first body part for detection.

[0156] The method for constructing a probe position prediction model in this embodiment predicts the actual positioning data of the actual detection probe entering into the first body part through the probe position prediction model to predict the actual detection probe corresponding to the input actual detection data, thereby achieving accurate and reliable positioning of the actual detection probe.

[0157] In an optional embodiment, if Figure 3 As shown, step S2 includes:

[0158] S22. According to the three-dimensional data corresponding to the three-dimensional body part model, simulate the virtual probe positioning data and virtual probe scanning data corresponding to the virtual probe for detecting the second body part.

[0159] The probe position prediction model is used to predict the actual positioning data of the actual detection probe corresponding to the input actual detection data, which is inserted into the body and outside the second body part.

[0160] Specifically, the second body part includes the liver, kidney, thyroid, breast or prostate. During the detection process, the actual detection probe can be inserted into the body and outside the second body part, that is, the actual detection probe can be in contact with the body skin, but detection is performed outside the second body part.

[0161] The method for constructing a probe position prediction model in this embodiment predicts the actual positioning data of the actual detection probe that enters the body and is outside the second body part through the probe position prediction model to predict the actual positioning data of the actual detection probe that enters the body and is outside the second body part, thereby achieving accurate and reliable positioning of the actual detection probe.

[0162] In an optional embodiment, if Figure 4 As shown, step S1 includes:

[0163] S11. Scan the target object based on the sample detection probe to collect sample detection data of the target object.

[0164] S12. Perform three-dimensional reconstruction processing on the sample detection data to establish a three-dimensional body part model.

[0165] The target subject (also called a patient) can be a human or an animal.

[0166] The sample detection data may be image data.

[0167] When the image data is an ultrasonic image of the heart, a detection probe can be used to perform an external ultrasonic scan of the patient. Specifically, when the patient lies flat, the detection probe is vertically pressed against the human body to collect an external two-dimensional image of the human heart from the neck to the waist. The heart sound data is monitored by a phonocardiograph, and the extracardiac ultrasonic probe is used to capture the ultrasonic image of the heart during the isovolumetric relaxation period. That is, the external two-dimensional image of the heart is screened for the third heart sound to obtain an ultrasonic image at the third heart sound. Several ultrasonic images at the third heart sound constitute the aforementioned sample detection data, and the sample detection data is a two-dimensional image.

[0168] The sample detection probe can be a 2D (two-dimensional) probe, such as Figure 5 As shown, a 2D probe scans to obtain a 2D image. At this time, the sample detection data is a two-dimensional image. The pixel nearest neighbor method (Pixel Nearest Neighbor, a pixel-based three-dimensional reconstruction algorithm) is used to perform three-dimensional reconstruction on several two-dimensional images to establish a three-dimensional body part model (i.e., a 3D model).

[0169] The sample detection probe can also be a 3D (three-dimensional) probe, such as Figure 6 As shown, the 3D data obtained by the 3D probe scanning includes several 2D images. At this time, the sample detection data is a two-dimensional image, and the partial three-dimensional body part model (i.e., 3D model) can be merged into a complete three-dimensional body part model (i.e., complete 3D model) based on voxel reconstruction, feature point alignment, and three-dimensional fusion.

[0170] In an optional embodiment, if Figure 7 As shown, step S2 includes:

[0171] S23, obtaining preset scanning data.

[0172] S24, simulating virtual probe positioning data corresponding to the virtual probe.

[0173] S25 . Obtain virtual probe scanning data based on the virtual probe positioning data and the preset scanning data.

[0174] In an optional embodiment, when the first body part includes a heart, before acquiring the preset scan data, the method further includes:

[0175] Obtain a preset position range of the virtual probe within the heart region.

[0176] The virtual probe positioning data is within the preset position range.

[0177] Below, a specific explanation is given by taking the first body part as the heart as an example.

[0178] The three-dimensional data of the three-dimensional body part model of the heart includes a bounding box B of the three-dimensional body part model of the heart=(X min ,X max ,Y min ,Y max ,Z min ,Z max ), the source point coordinates O = (X, Y, Z).

[0179] The source coordinates of the three-dimensional body model of the heart are O = (X, Y, Z), X min is the minimum value of the three-dimensional body part model of the heart on the X-axis in its coordinates, X max is the maximum value of the three-dimensional body part model of the heart on the X axis in its coordinates, min is the minimum value of the three-dimensional body part model of the heart on the Y axis in its coordinates, max is the maximum value of the three-dimensional body part model of the heart on the Y axis in its coordinates, min is the minimum value of the three-dimensional body part model of the heart on the Z axis in its coordinates, max It is the maximum value of the three-dimensional body part model of the heart on the Z axis in its coordinates.

[0180] According to the following formula:

[0181]

[0182] The source point coordinate O of the three-dimensional body part model of the heart is adjusted to the new source point coordinate O′ in the world coordinate system, so that the geometric center of the three-dimensional data of the three-dimensional body part model of the heart is aligned with the source point coordinate O′=(0,0,0) in the world coordinate system. At the same time, all coordinates in the three-dimensional data will also be displaced and adjusted accordingly, and the displacement adjustment method is the same as the movement method of the source point coordinate O.

[0183] The world coordinate system is a reference frame used to describe the position and orientation of objects. It is often used in the field of computer graphics and 3D models to ensure consistent rendering and positioning of objects in different environments. The world coordinate system consists of three mutually perpendicular axes (usually X, Y, and Z axes), each of which represents an independent spatial direction. By giving coordinate values, the position of an object relative to the world coordinate system can be accurately located and oriented. In this way, different objects can be accurately compared and interacted with based on their positions in the world coordinate system.

[0184] At this time, the actual detection probe is an intracardiac ultrasound probe, which can be built into an ICE (intracardiac ultrasound) catheter. When collecting, the ICE catheter is inserted into the heart cavity through peripheral vascular intervention, and the intracardiac ultrasound probe can be used to collect a two-dimensional image of the endocardium. The positions of the ICE probe (i.e., the actual detection probe) in the heart are mainly:

[0185] 1) Right atrium: The ICE probe is inserted into the heart through the femoral vein, first entering the middle of the right atrium from the inferior vena cava. By rotating the ICE catheter, the internal structures of the heart, such as the right ventricle, left atrium, fossa ovale, and pulmonary veins, can be observed. This can guide surgeries such as atrial septal puncture, foramen ovale closure, and screening for atrial thrombus.

[0186] 2) Left atrium: When the ICE probe is in the right atrium, after making the A / P bend and delivering the catheter, it can enter the left atrium along the oval fossa. The ICE catheter in the left atrium can guide the left atrial appendage occlusion surgery.

[0187] 3) Right ventricle: When the ICE probe is in the right atrium, it can enter the right ventricle along the direction of the tricuspid valve after making the A / P bend and delivering the catheter. The ICE catheter in the right ventricle can guide the ablation process of arrhythmias by constructing the long axis of the left ventricle, constructing the short axis of the right ventricle, and monitoring the pericardial effusion at the bottom of the right ventricle.

[0188] According to the usage scenario of ICE catheter in cardiac surgery, we define set I = I RA ∪I LA ∪I RV is the preset position range (value range) of the ICE probe in the heart area, where I RA is the location range of the right atrium, I LA is the position range of the left atrium, I RVIt is the location range of the right ventricle.

[0189] For example, point O (0,0,0) is used as the source point, and point A (400,400,0) and point B (400,-400,0) are used as the radius endpoints to construct the initial virtual sector F of the space. The data corresponding to the initial virtual sector F is the preset scanning data. The actual values ​​of point A and point B can be set according to the scanning range of different models of intracardiac ultrasound probes.

[0190] The virtual probe is composed of a probe model (here as a point) and a transformation matrix. The origin of the virtual probe is on the imaging device. Similarly, the initial virtual sector is composed of a sector model and a transformation matrix. Among them, the transformation matrix of the virtual probe is always equal to the transformation matrix of the initial virtual sector, both of which are M. The virtual probe is a point, and its position and orientation are defined by the transformation matrix M (i.e., virtual probe positioning data). According to the initial virtual sector F and the virtual probe positioning data M, the cross-sectional data F′ (i.e., virtual probe scanning data) scanned by the virtual probe in the three-dimensional organ model of the heart is obtained.

[0191] The initial virtual sector F is subjected to position transformation processing by means of the transformation matrix M. The position transformation processing includes but is not limited to rotation and movement operations, so as to obtain virtual probe scanning data F′ after position transformation of the initial virtual sector F, where F′=MF.

[0192] Among them, the virtual probe positioning data Characterizes the position and orientation of the virtual probe, S = (x, y, z) is the position coordinate of the virtual probe, is the rotation matrix of the initial virtual sector F (i.e. the orientation of the virtual probe), a 11 ~a 33 is the orientation / angle information of the virtual probe, and the position coordinates of the virtual probe are within the above-mentioned position range I, that is, S(x, y, z)∈I.

[0193] In an optional embodiment, if Figure 8 As shown, step S3 includes:

[0194] S31. For any preset sampling point of the virtual probe in the three-dimensional body part model, based on the intersection of the virtual probe scanning data corresponding to the preset sampling point and the three-dimensional body part model, obtain initial three-dimensional intersection data corresponding to the preset sampling point.

[0195] S32, based on the virtual probe positioning data corresponding to the preset sampling point and the initial three-dimensional intersection data, obtaining the initial two-dimensional structure data corresponding to the preset sampling point.

[0196] S33, obtaining a random transformation matrix corresponding to a preset sampling point.

[0197] S34. Obtain a two-dimensional structure sample data set based on a number of random transformation matrices and corresponding initial two-dimensional structure data.

[0198] In the process of obtaining a two-dimensional structure sample data set, it is necessary to obtain a random transformation matrix and initial two-dimensional structure data corresponding to a number of preset sampling points.

[0199] Taking the three-dimensional body part model of the heart as an example, for example, for a preset sampling point i, according to the virtual probe scanning data F corresponding to the preset sampling point i i The intersection of the three-dimensional body part model of the heart is obtained to obtain the initial three-dimensional intersection data P corresponding to the preset sampling point i i Then use the virtual probe to locate the data M i , find M i The inverse fan transformation matrix M i ′,M i ′=M i -1 , according to the formula: P i ′=RM i -1 P i , and finally get the initial three-dimensional intersection data P i The corresponding two-dimensional expression P i ′ (i.e., initial two-dimensional structure data). The image schematic diagram corresponding to the initial two-dimensional structure data is as follows: Fig. 9 shown.

[0200] Where R is the transformation matrix, The 400 in R corresponds to the 400 in the initial virtual sector F of the space constructed with point A (400, 400, 0) and point B (400, -400, 0) as the radius endpoints, that is, the transformation matrix R is related to the radius endpoints in the initial virtual sector F.

[0201] A two-dimensional structure sample data set is obtained according to the random transformation matrix of a number of preset sampling points i and the corresponding initial two-dimensional structure data.

[0202] Through the aforementioned calculations, the initial two-dimensional structural data of the virtual probe in the three-dimensional body part model is obtained to simulate the actual detection data collected by the actual detection probe.

[0203] In an optional implementation, the step of obtaining a random transformation matrix corresponding to a preset sampling point includes:

[0204] Get the random rotation matrix corresponding to the preset sampling point;

[0205] Get the target position coordinates corresponding to the preset sampling points;

[0206] Based on the random rotation matrix and the target position coordinates, a random transformation matrix is ​​obtained.

[0207] The rotation and displacement of the virtual probe are independent of each other and can take values ​​separately. The rotation order of the x-axis, y-axis and z-axis of the virtual probe is interdependent. The position coordinates of the virtual probe are constrained to the aforementioned position range I. The x, y and z positions of the virtual probe are independent of each other and can take values ​​separately under the constraints.

[0208] Specifically, random sampling of samples can be carried out according to the following steps:

[0209] 1) Randomly arrange the rotation order of the x-axis, y-axis, and z-axis. Each rotation method θ randomly takes a value in [0,2π), so the random rotation matrix can be obtained The x-axis rotation matrix y-axis rotation matrix z-axis rotation matrix θx, θy, θz represent the rotation angles of the virtual probe on the x-axis, y-axis, and z-axis respectively. The values ​​of θx, θy, and θz are randomly selected in [0,2π). W3, W2, and W1 are the matrices corresponding to the three rotation operations. The values ​​of W3, W2, and W1 are taken from W x , W y , W z , and only one of the x-axis, y-axis, and z-axis is selected, and the rotation order is not limited, that is, if W3 is rotated from W x If W2 can only be selected from W y or W z If W2 is taken from W z If W1 can only be taken from W y Take the value from.

[0210] 2) Randomly select the area where the virtual probe is located in the body part (i.e., the target area), so that the target area D i In the body part area, that is, D i ∈I, I corresponds to the preset position range of the aforementioned virtual probe in the body part area.

[0211] 3) The target position coordinates S of the virtual probe are i =(x,y,z) to get a random value, where The target position coordinates are within the target area.

[0212] 4) Obtain the random transformation matrix through random sampling steps 1) to 3) S i T YesS i The transposed matrix of the initial two-dimensional structure data P′ is obtained based on i , and finally get the random position structure sample pair (W i , Pi ′).

[0213] According to the random transformation matrix of all preset sampling points and the corresponding initial two-dimensional structure data, a two-dimensional structure sample data set is obtained, that is, all random position structure sample pairs (W i , P i ′) constitutes a two-dimensional structure sample data set. For example, 1000 preset sampling points are sampled to obtain 1000 random position structure sample pairs to form a two-dimensional structure sample data set. The value range of the preset sampling points is 1 to 1000.

[0214] The two-dimensional structure sample data set can also be called a two-dimensional contour point set, such as Fig.10 As shown in the figure, if the virtual probe is a virtual 2D probe, the virtual 2D probe corresponds to a 2D cross section, which corresponds to a two-dimensional structure sample data set (i.e., a contour point set). If the virtual probe is a virtual 3D probe, the virtual 3D probe corresponds to 3D data containing several 2D cross sections, which ultimately corresponds to a two-dimensional structure sample data set (i.e., a contour point set), and then the probe position prediction model is trained.

[0215] The actual body part includes a first body part that can be detected by the actual detection probe, or a second body part that cannot be detected by the actual detection probe; the first body part includes the heart or cardiovascular system, and the second body part includes the liver, kidney, thyroid gland, breast or prostate gland. The above takes the first body part as the heart as an example to introduce in detail the process of obtaining the two-dimensional structure sample data set corresponding to the heart. The process of obtaining the two-dimensional structure sample data set corresponding to other actual body parts is similar to the process of obtaining the two-dimensional structure sample data set corresponding to the heart, and will not be repeated here.

[0216] Specifically, the detection probe includes an ultrasonic probe, and other types of detection probes may also be used to detect body parts as long as corresponding data acquisition and functions can be achieved.

[0217] In an optional implementation, the step of training a probe position prediction model according to a two-dimensional structure sample data set includes:

[0218] The preset network model is trained based on the two-dimensional structure sample data set to obtain the probe position prediction model.

[0219] In an optional implementation, the preset network model is constructed based on the Encoder structure model and the Decoder structure model.

[0220] Specifically, the preset network model is built based on the UNet network and the PointNet++ network.

[0221] The UNet network is used to obtain multi-scale feature information (layer feature information) through upsampling links and downsampling links; at the same time, the PointNet++ network is used to process the image point cloud data in the input sample data set to obtain the image position information.

[0222] In an optional embodiment, the probe position prediction model includes an input layer, a downsampling layer, an upsampling layer, and an output layer;

[0223] The steps of training a preset network model based on a two-dimensional structure sample data set to obtain a probe position prediction model include:

[0224] Input the two-dimensional structure sample data set into the input layer for sampling operation, and output the number of initial actual point sets corresponding to the initial two-dimensional structure data;

[0225] If the number of initial actual point sets is greater than the number of preset point sets, the initial two-dimensional structure data is input to the downsampling layer, and the target two-dimensional structure data is output;

[0226] If the number of initial actual point sets is less than the preset number of point sets, the initial two-dimensional structure data is input to the upsampling layer, and the target two-dimensional structure data is output;

[0227] The number of actual target point sets corresponding to the target two-dimensional structure data is equal to the number of preset point sets;

[0228] The target two-dimensional structure data is input to the output layer, and the actual positioning data when the actual detection probe detects the actual body part is output.

[0229] In the input layer, it is necessary to perform sampling operations on the two-dimensional structure sample data set so that the number of initial actual point sets in the initial two-dimensional structure data of each preset sampling point is the preset number of point sets. The downsampling layer uses a multi-layer perceptron and the farthest point sampling method to process the initial two-dimensional structure data and output the target two-dimensional structure data so that the number of target actual point sets corresponding to the target two-dimensional structure data is equal to the preset number of point sets; the upsampling layer uses a multi-layer perceptron and the contour interpolation method to process the initial two-dimensional structure data and output the target two-dimensional structure data so that the number of target actual point sets corresponding to the target two-dimensional structure data is equal to the preset number of point sets.

[0230] For example, the preset network model is a LNN (Localization Neural Network) model, and a probe position prediction model is obtained by training the LNN model.

[0231] For example, the model structure diagram of the probe position prediction model is as follows: Fig.11As shown in the figure, the multi-layer perceptrons of the down sampling layer and the up sampling layer both use a three-layer structure. The target two-dimensional structure data includes layer feature information and image position information, which realizes the processing of image point cloud data in the sample data set and the extraction of layer feature information, thereby improving the accuracy of the model.

[0232] For example, if the number of preset point sets is 1024, and the initial two-dimensional structure data P′ of the preset sampling point i is i The number of initial actual point sets in is greater than 1024, then P i ′=s d (P i ), if P i If the number of points is less than 1024, then P i ′=s u (P i ), where s d (h) is the farthest point sampling method, s u (h) is the contour interpolation method. The downsampling layer mainly includes MLP (Multi-layer Perceptron) and the farthest point sampling method, and the upsampling layer mainly includes MLP and the contour interpolation method. Assume that e is the upper layer input, f(e) is MLP, For the concat operation, the output formula of the downsampling layer can be expressed as The output formula of the upsampling layer can be expressed as Finally, the output layer uses a fully connected layer to transform the feature tensor t (1024×3×64×N) Convert to output tensor t (12×N) , that is, the actual positioning data, where N is the batch size in the training process, indicating the number of samples used in each iterative training.

[0233] The actual positioning data corresponds to the x, y, z displacement information of the probe and the a 11 ~a 33 Angle information.

[0234] The SmoothL1 loss function used by the preset network model is mainly composed of two parts. One part is composed of the x, y, z displacement information in the position coordinates of the virtual probe, and the other part is composed of the a 11 ~a 33 The loss function is composed of angle information and is expressed as follows:

[0235]

[0236] in is the mean displacement, is the angle mean, k is a sample, a k For a sample 11 ~a 33 The mean of N is the batch size in the aforementioned training process, and i and j are the angle information a 11 ~a 33 The subscript in .

[0237] The method for constructing a probe position prediction model in this embodiment can accurately predict the actual detection probe corresponding to the input actual detection data and the actual positioning data when detecting the actual body part through the probe position prediction model, thereby achieving accurate and reliable positioning of the actual detection probe.

[0238] Example 2

[0239] This embodiment provides a probe position prediction method, which includes the method for constructing a probe position prediction model in Embodiment 1, such as Fig.12 As shown, the probe position prediction method includes:

[0240] S5. Based on the initial detection data collected by the actual detection probe, obtain target detection data corresponding to the actual body part.

[0241] S6. Input the target detection data into the probe position prediction model obtained based on the construction method to output the actual positioning data when the actual detection probe detects the actual body part.

[0242] In an optional embodiment, if Fig.13 As shown, step S5 includes:

[0243] S51. Acquire initial detection data collected by an actual detection probe.

[0244] S52: Perform semantic segmentation operation on the initial detection data to obtain a number of segmentation mask images.

[0245] S53, performing contour extraction operation on a plurality of segmentation mask images to obtain target detection data.

[0246] The actual detection probe can be a 2D probe or a 3D probe; Fig.14 As shown in the figure, 2D probe scanning obtains 2D images, that is, the corresponding initial detection data is a two-dimensional image; 3D probe scanning obtains 3D data containing several 2D images, that is, the corresponding initial detection data is also a two-dimensional image. After semantic segmentation and contour extraction operations, the contour point set (that is, target detection data) can be obtained, and then input into the probe position prediction model for prediction (inference).

[0247] Taking the actual body part as the heart as an example, the initial detection data corresponding to the heart can be an initial intracardiac two-dimensional ultrasound image. Fig.15 shown.

[0248] In an optional implementation, the step of performing a semantic segmentation operation on the initial detection data to obtain a plurality of segmentation mask images includes:

[0249] Based on the preset semantic segmentation model, a semantic segmentation extraction operation is performed on the initial detection data to obtain several segmentation mask images.

[0250] The preset semantic segmentation model can be obtained by training the DeepLab V3 Plus (a semantic segmentation model) model. Taking the heart as an internal organ as an example, for example, 5,000 intracardiac ultrasound label samples are used to train the DeepLab V3 Plus model to obtain the preset semantic segmentation model, and the initial intracardiac two-dimensional ultrasound image is input into the preset semantic segmentation model to obtain a segmentation mask image of the initial intracardiac two-dimensional ultrasound image.

[0251] The calculation of the preset semantic segmentation model can be expressed as:

[0252]

[0253] Where b represents the serial number of the image pixel, each serial number represents the two-dimensional coordinate position in an image, T represents the number of pixels in the image, p(b) represents the annotation type of the pixel, β represents the trainable parameters in the preset semantic segmentation model, and p β (b; β) indicates the type of the b-th pixel point calculated under the current parameter β. After multiple rounds of iterative calculations, the optimal solution β is obtained. * .

[0254] By calculating the preset semantic segmentation model, the segmentation mask image of the heart tissue area can be obtained. The schematic diagram of the segmentation mask image is as follows: Fig.16 shown.

[0255] In an optional implementation, the step of performing contour extraction operations on a plurality of segmentation mask images to obtain target detection data includes:

[0256] Traverse all foreground pixels in the segmentation mask image to obtain initial edge contour points;

[0257] Traversing the initial edge contour points, determining whether each target edge contour point in the initial edge contour points is a preset contour point;

[0258] If so, the target edge contour points are removed from the initial edge contour points, and the remaining edge contour points after removing the target edge contour points from the initial edge contour points are used as target detection data.

[0259] When the detection probe is an ultrasonic probe, the preset contour points are preset sector contour points.

[0260] By traversing all foreground pixels Q in the segmentation mask image i , determine whether the point is an initial edge contour point by the following method;

[0261]

[0262] Among them, the pixel value of all foreground pixels in the segmentation mask image is 1, the pixel value of the background pixel is 0, v represents the sequence number of the image pixel, each sequence number corresponds to a two-dimensional coordinate position in the segmentation mask image, G8(v) represents all points in the 8-neighborhood of the vth point (excluding v), Sum(·) represents the sum of the pixel values ​​of all foreground pixels, if Q i =1 indicates that the foreground pixel is a contour point (ie, an initial edge contour point), otherwise it is not an initial edge contour point.

[0263] Taking the body part as the heart and the detection probe as an ultrasound probe as an example, the schematic diagram of the contour map formed by the initial edge contour points is as follows Fig.17 As shown, it can be seen that in some heart tissue areas, part of the contour is replaced by the contour of the fan because it is at the edge of the fan, and the contour of the fan is not the real contour of the tissue, so these contours from the fan need to be removed.

[0264] Ultrasound images produced by ultrasound machines of the same model correspond to the same sector area. After the sector area is determined, the corresponding sector mask image can be obtained, and then the preset sector contour points can be determined. Specifically, the sector area and the interior in the sector mask image are the foreground (pixel value is 1), and the area outside the sector area is the background (pixel value is 0). The following formula is used to determine whether a point in the sector area is a preset contour point.

[0265]

[0266] Among them, the pixel value of all foreground pixels in the sector mask image is 1, the pixel value of the background pixel is 0, v represents the sequence number of the image pixel, each sequence number corresponds to a two-dimensional coordinate position in the sector mask image, G8(v) represents all points in the 8-neighborhood of the vth point (excluding v), Sum(·) represents the sum of the pixel values ​​of all foreground pixels, if S i =1 indicates that the point is a preset contour point.

[0267] Compare the initial contour point Q i Is it a preset contour point S? i(i.e., the contour of the fan). If it is a fan contour, it needs to be removed from the initial edge contour points. At this time, the foreground is all areas within the fan, and the background is all areas outside the fan. After calculation, the tissue contour map (i.e., target detection data) with the fan edge removed can be obtained. The schematic diagram of the target detection data is shown in Fig.18 shown.

[0268] The probe position prediction method of this embodiment is based on the probe position prediction model in Embodiment 1, predicts the actual detection probe corresponding to the input actual detection data, and the actual positioning data when detecting the actual body part, thereby achieving accurate and reliable positioning of the actual detection probe.

[0269] Example 3

[0270] This embodiment provides a system for constructing a probe position prediction model. Fig.19 As shown, the build system includes:

[0271] A model building module 1, used to build a three-dimensional body part model corresponding to the sample body part based on the sample detection data corresponding to the sample body part;

[0272] A virtual probe simulation module 2 is used to simulate virtual probe positioning data and virtual probe scanning data corresponding to the virtual probe according to the three-dimensional data corresponding to the three-dimensional body part model;

[0273] The sample acquisition module 3 is used to acquire a two-dimensional structure sample data set corresponding to the sample body part based on the virtual probe positioning data and the virtual probe scanning data;

[0274] Model training module 4, used for training a probe position prediction model according to a two-dimensional structure sample data set;

[0275] Among them, the probe position prediction model is used to predict the actual detection probe corresponding to the input actual detection data, and the actual positioning data when detecting the actual body part; the actual body part includes the first body part that the actual detection probe can detect, or the second body part that the actual detection probe cannot detect.

[0276] In an optional embodiment, the virtual probe simulation module 2 is used to simulate the virtual probe positioning data corresponding to the virtual probe detecting the first body part and the virtual probe scanning data according to the three-dimensional data corresponding to the three-dimensional body part model;

[0277] The probe position prediction model is used to predict actual positioning data of an actual detection probe corresponding to the input actual detection data, which is probed into the first body part.

[0278] In an optional embodiment, the virtual probe simulation module 2 is used to simulate the virtual probe positioning data corresponding to the virtual probe for detecting the second body part and the virtual probe scanning data according to the three-dimensional data corresponding to the three-dimensional body part model;

[0279] The probe position prediction model is used to predict the actual positioning data of the actual detection probe corresponding to the input actual detection data, which is inserted into the body and outside the second body part.

[0280] In an optional embodiment, the model building module 1 includes:

[0281] The data acquisition unit 11 is used to scan the target object based on the sample detection probe to acquire sample detection data of the target object;

[0282] The model building unit 12 is used to perform three-dimensional reconstruction processing on the sample detection data to build a three-dimensional body part model.

[0283] In an optional implementation, the virtual probe simulation module 2 includes:

[0284] A preset data acquisition unit 21, used to acquire preset scanning data;

[0285] A positioning data acquisition unit 22, used for simulating virtual probe positioning data corresponding to the virtual probe;

[0286] The scanning data acquisition unit 23 is used to obtain virtual probe scanning data based on the virtual probe positioning data and the preset scanning data.

[0287] In an optional embodiment, when the first body part includes a heart, the virtual probe simulation module 2 further includes:

[0288] A position range acquisition unit 20, used to acquire a preset position range of the virtual probe in the heart area;

[0289] The virtual probe positioning data is within the preset position range.

[0290] In an optional implementation, the sample acquisition module 3 includes:

[0291] A three-dimensional intersection point acquisition unit 31 is used to obtain initial three-dimensional intersection point data corresponding to any preset sampling point of the virtual probe in the three-dimensional body part model based on the intersection of the virtual probe scanning data corresponding to the preset sampling point and the three-dimensional body part model;

[0292] The initial data acquisition unit 32 is used to obtain the initial two-dimensional structure data corresponding to the preset sampling point based on the virtual probe positioning data corresponding to the preset sampling point and the initial three-dimensional intersection data;

[0293] A transformation matrix acquisition unit 33, used to acquire a random transformation matrix corresponding to a preset sampling point;

[0294] The sample data construction unit 34 is used to obtain a two-dimensional structure sample data set based on a number of random transformation matrices and corresponding initial two-dimensional structure data.

[0295] In an optional implementation, the transformation matrix acquisition unit 33 is further used to acquire a random rotation matrix corresponding to a preset sampling point; acquire target position coordinates corresponding to the preset sampling point; and obtain a random transformation matrix based on the random rotation matrix and the target position coordinates.

[0296] In an alternative embodiment, the first body part includes the heart or cardiovascular system, and the second body part includes the liver, kidney, thyroid gland, breast, or prostate gland.

[0297] In an optional embodiment, the detection probe includes an ultrasound probe.

[0298] In an optional implementation, the model training module 4 is used to train a preset network model based on a two-dimensional structure sample data set to obtain a probe position prediction model.

[0299] In an optional implementation, the preset network model is constructed based on the Encoder structure model and the Decoder structure model.

[0300] In an optional implementation, the preset network model is constructed based on the UNet network and the PointNet++ network.

[0301] In an optional embodiment, the probe position prediction model includes an input layer, a downsampling layer, an upsampling layer and an output layer; the model training module 4 is used to input the two-dimensional structure sample data set into the input layer for sampling operation, and output the initial actual point set number corresponding to the initial two-dimensional structure data; if the initial actual point set number is greater than the preset point set number, the initial two-dimensional structure data is input into the downsampling layer, and the target two-dimensional structure data is output; if the initial actual point set number is less than the preset point set number, the initial two-dimensional structure data is input into the upsampling layer, and the target two-dimensional structure data is output; wherein the target actual point set number corresponding to the target two-dimensional structure data is equal to the preset point set number; the target two-dimensional structure data is input into the output layer, and the actual positioning data when the actual detection probe detects the actual body part is output.

[0302] In an optional implementation, the model training module 4 uses a multi-layer perceptron and a farthest point sampling method to process the initial two-dimensional structure data and output the target two-dimensional structure data;

[0303] The target two-dimensional structure data includes layer feature information and image position information.

[0304] In an optional implementation, the model training module 4 uses a multi-layer perceptron and contour interpolation method to process the initial two-dimensional structure data and output the target two-dimensional structure data;

[0305] The target two-dimensional structure data includes layer feature information and image position information.

[0306] The system for constructing the probe position prediction model of this embodiment corresponds to the method for constructing the probe position prediction model in Embodiment 1. The working principles of the system and the method are the same, and will not be described in detail here.

[0307] The system for constructing the probe position prediction model of the present embodiment establishes a three-dimensional body part model corresponding to the sample body part based on the sample detection data corresponding to the sample body part, simulates the virtual probe positioning data and the virtual probe scanning data corresponding to the virtual probe according to the three-dimensional data corresponding to the three-dimensional body part model, and finally obtains the two-dimensional structure sample data set corresponding to the sample body part, and then trains the probe position prediction model based on the two-dimensional structure sample data set, and predicts the actual detection probe corresponding to the input actual detection data through the probe position prediction model, and the actual positioning data when detecting the actual body part, thereby realizing accurate and reliable positioning of the actual detection probe.

[0308] Example 4

[0309] This embodiment provides a probe position prediction system, which includes a construction system of a probe position prediction model in Embodiment 3, such as Fig. 20 As shown, the probe position prediction system includes:

[0310] The detection data acquisition module 5 is used to obtain the target detection data corresponding to the actual body part based on the initial detection data collected by the actual detection probe;

[0311] The probe position prediction module 6 is used to input the target detection data into the probe position prediction model obtained based on the construction system to output the actual positioning data when the actual detection probe detects the actual body part.

[0312] In an optional implementation, the detection data acquisition module 5 includes:

[0313] An initial data acquisition unit 51 is used to acquire initial detection data actually acquired by the detection probe;

[0314] A semantic segmentation unit 52, used for performing a semantic segmentation operation on the initial detection data to obtain a plurality of segmentation mask images;

[0315] The contour extraction unit 53 is used to perform contour extraction operations on a plurality of segmentation mask images to obtain target detection data.

[0316] In an optional implementation, the semantic segmentation unit 52 is used to perform a semantic segmentation operation on the initial detection data based on a preset semantic segmentation model to obtain a plurality of segmentation mask images.

[0317] In an optional embodiment, the contour extraction unit 53 is used to traverse all foreground pixel points in the segmentation mask image to obtain initial edge contour points; traverse the initial edge contour points to determine whether each target edge contour point in the initial edge contour points is a preset contour point; if so, remove the target edge contour points from the initial edge contour points, and use the remaining edge contour points after removing the target edge contour points from the initial edge contour points as target detection data.

[0318] The probe position prediction system of this embodiment corresponds to the probe position prediction method in Embodiment 2. The working principles of the probe position prediction system and the probe position prediction method are the same, and will not be described in detail here.

[0319] The probe position prediction system of this embodiment can accurately predict the actual detection probe corresponding to the actual detection data input, and the actual positioning data when detecting the actual body part, through the probe position prediction model in Example 3, to achieve accurate and reliable positioning of the actual detection probe.

[0320] Example 5

[0321] This embodiment provides a probe device, which includes a detection probe and the probe position prediction system in Embodiment 4.

[0322] The probe device may also include other components.

[0323] The probe device of this embodiment, through the probe position prediction system in Example 4, can accurately predict the actual detection probe corresponding to the input actual detection data and the actual positioning data when detecting the actual body part, thereby achieving accurate and reliable positioning of the detection probe.

[0324] Example 6

[0325] This embodiment provides an electronic device, Fig.21 A structural schematic diagram of an electronic device provided for this embodiment, the electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor, and when the processor executes the computer program, it implements the method for constructing the probe position prediction model in the above-mentioned embodiment 1, or the probe position prediction method in the above-mentioned embodiment 2. Fig.21The electronic device 70 shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure. Fig.21 As shown, the electronic device 70 may be in the form of a general-purpose computing device, for example, it may be a server device. The components of the electronic device 70 may include, but are not limited to: at least one processor 71, at least one memory 72, and a bus 73 connecting different system components (including the memory 72 and the processor 71).

[0326] The bus 73 includes a data bus, an address bus, and a control bus.

[0327] The memory 72 may include a volatile memory, such as a random access memory (RAM) 721 and / or a cache memory 722 , and may further include a read-only memory (ROM) 723 .

[0328] The memory 72 may also include a program tool 725 (or utility) having a set (at least one) of program modules 724, such program modules 724 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0329] The processor 71 executes various functional applications and data processing by running the computer program stored in the memory 72, such as the method for constructing the probe position prediction model in the above-mentioned embodiment 1, or the probe position prediction method in the above-mentioned embodiment 2.

[0330] The electronic device 70 may also communicate with one or more external devices 74. Such communication may be performed via an input / output (I / O) interface 75. Furthermore, the model generating electronic device 70 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 76. Fig.21 As shown, the network adapter 76 communicates with other modules of the electronic device 70 via the bus 73. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 70, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.

[0331] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0332] Example 7

[0333] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the method for constructing a probe position prediction model in the above-mentioned embodiment 1, or the method for predicting the probe position in the above-mentioned embodiment 2.

[0334] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.

[0335] In a possible implementation, the present disclosure can also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps in the method for constructing the probe position prediction model in the above-mentioned embodiment 1, or the steps in the method for predicting the probe position in the above-mentioned embodiment 2.

[0336] Among them, the program code for executing the present disclosure can be written in any combination of one or more programming languages, and the program code can be executed completely on the user device, partially on the user device, as a separate software package, partially on the user device and partially on a remote device, or completely on the remote device.

[0337] Although the specific embodiments of the present disclosure are described above, those skilled in the art should understand that this is only an example, and the protection scope of the present disclosure is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present disclosure, but these changes and modifications all fall within the protection scope of the present disclosure.

Claims

1. A method for constructing a probe position prediction model, characterized in that: The construction method comprises: Based on the sample detection data corresponding to the sample body part, establishing a three-dimensional body part model corresponding to the sample body part; Simulating virtual probe positioning data and virtual probe scanning data corresponding to the virtual probe according to the three-dimensional data corresponding to the three-dimensional body part model; Based on the virtual probe positioning data and the virtual probe scanning data, obtaining a two-dimensional structure sample data set corresponding to the sample body part; The probe position prediction model is obtained by training according to the two-dimensional structure sample data set; The probe position prediction model is used to predict the actual positioning data of the actual detection probe corresponding to the input actual detection data when detecting the actual body part; The actual body part includes a first body part that the actual detection probe can detect, or a second body part that the actual detection probe cannot detect; The step of acquiring a two-dimensional structure sample data set corresponding to the sample body part based on the virtual probe positioning data and the virtual probe scanning data comprises: For any preset sampling point of the virtual probe in the three-dimensional body part model, based on the intersection of the virtual probe scanning data corresponding to the preset sampling point and the three-dimensional body part model, obtaining initial three-dimensional intersection data corresponding to the preset sampling point; Based on the virtual probe positioning data corresponding to the preset sampling point and the initial three-dimensional intersection data, obtaining the initial two-dimensional structure data corresponding to the preset sampling point; Obtaining a random transformation matrix corresponding to the preset sampling point; Obtaining the two-dimensional structure sample data set based on a plurality of the random transformation matrices and the corresponding initial two-dimensional structure data; The step of obtaining the random transformation matrix corresponding to the preset sampling point includes: Obtaining a random rotation matrix corresponding to the preset sampling point; Obtaining the target position coordinates corresponding to the preset sampling point; Based on the random rotation matrix and the target position coordinates, obtaining the random transformation matrix; The step of obtaining the probe position prediction model through training according to the two-dimensional structure sample data set comprises: Training a preset network model based on the two-dimensional structure sample data set to obtain the probe position prediction model; The probe position prediction model includes an input layer, a downsampling layer, an upsampling layer and an output layer; The step of training a preset network model based on the two-dimensional structure sample data set to obtain the probe position prediction model includes: Inputting the two-dimensional structure sample data set into the input layer for sampling operation, and outputting the number of initial actual point sets corresponding to the initial two-dimensional structure data; If the number of the initial actual point set is greater than the number of the preset point set, the initial two-dimensional structure data is input to the downsampling layer, and the target two-dimensional structure data is output; If the number of the initial actual point set is less than the number of the preset point set, inputting the initial two-dimensional structure data to the upsampling layer, and outputting the target two-dimensional structure data; The number of actual target point sets corresponding to the target two-dimensional structure data is equal to the number of preset point sets; The target two-dimensional structure data is input into the output layer, and the actual positioning data when the actual detection probe detects the actual body part is output.

2. The construction method according to claim 1, characterized in that: The step of simulating virtual probe positioning data corresponding to the virtual detection probe and virtual probe scanning data according to the three-dimensional data corresponding to the three-dimensional body part model comprises: Simulating the virtual probe positioning data corresponding to the virtual probe detecting the first body part and the virtual probe scanning data according to the three-dimensional data corresponding to the three-dimensional body part model; The probe position prediction model is used to predict the actual positioning data of the actual detection probe corresponding to the input actual detection data, which is probed into the first body part; or, simulating the virtual probe positioning data and the virtual probe scanning data corresponding to the virtual probe for detecting the second body part according to the three-dimensional data corresponding to the three-dimensional body part model; The probe position prediction model is used to predict the actual positioning data of the actual detection probe corresponding to the input actual detection data, which is inserted into the body and outside the second body part.

3. The construction method according to claim 1, characterized in that: The step of establishing a three-dimensional body part model corresponding to the sample body part based on the sample detection data corresponding to the sample body part comprises: Scanning the target object based on the sample detection probe to collect sample detection data of the target object; The sample detection data is subjected to three-dimensional reconstruction processing to establish the three-dimensional body part model.

4. The construction method according to claim 1, characterized in that: The step of simulating virtual probe positioning data corresponding to the virtual probe and virtual probe scanning data according to the three-dimensional data corresponding to the three-dimensional body part model comprises: Get preset scan data; Simulating the virtual probe positioning data corresponding to the virtual probe; The virtual probe scanning data is obtained based on the virtual probe positioning data and the preset scanning data.

5. The construction method according to claim 4, characterized in that: When the first body part includes a heart, the step of obtaining preset scan data further includes: Acquiring a preset position range of the virtual probe in the heart area; Wherein, the virtual probe positioning data is located within the preset position range.

6. The construction method according to any one of claims 1 to 5, characterized in that: The first body part includes the heart or cardiovascular system; the second body part includes the liver, kidney, thyroid gland, breast or prostate gland; And / or, the detection probe includes an ultrasonic probe.

7. The construction method according to claim 1, characterized in that: The preset network model is constructed based on the Encoder structure model and the Decoder structure model.

8. The construction method according to claim 1, characterized in that: The preset network model is constructed based on the UNet network and the PointNet++ network.

9. The construction method according to claim 1, characterized in that: The step of inputting the initial two-dimensional structure data into the downsampling layer and outputting the target two-dimensional structure data comprises: Using a multi-layer perceptron and a farthest point sampling method, the initial two-dimensional structure data is processed to output the target two-dimensional structure data; Wherein, the target two-dimensional structure data includes layer feature information and image position information; and / or, The step of inputting the initial two-dimensional structure data into the upsampling layer and outputting the target two-dimensional structure data comprises: Using a multi-layer perceptron and contour interpolation method to process the initial two-dimensional structure data, and output the target two-dimensional structure data; The target two-dimensional structure data includes layer feature information and image position information.

10. A probe position prediction method, characterized in that: The probe position prediction method comprises a method for constructing a probe position prediction model according to any one of claims 1 to 9, and the probe position prediction method comprises: Based on the initial detection data collected by the actual detection probe, target detection data corresponding to the actual body part is obtained; The target detection data is input into the probe position prediction model obtained based on the construction method to output the actual positioning data when the actual detection probe detects the actual body part.

11. The probe position prediction method according to claim 10, characterized in that: The step of obtaining target detection data based on the initial detection data collected by the actual detection probe includes: Acquiring the initial detection data collected by the actual detection probe; Perform semantic segmentation operations on the initial detection data to obtain several segmentation mask images; Performing contour extraction operation on a plurality of the segmentation mask images to obtain the target detection data.

12. The probe position prediction method according to claim 11, characterized in that: The step of performing a semantic segmentation operation on the initial detection data to obtain a plurality of segmentation mask images comprises: Performing a semantic segmentation operation on the initial detection data based on a preset semantic segmentation model to obtain a plurality of segmentation mask images; and / or, The step of performing contour extraction operation on the plurality of segmentation mask images to obtain the target detection data comprises: Traversing all foreground pixels in the segmentation mask image to obtain initial edge contour points; Traversing the initial edge contour points, and determining whether each target edge contour point in the initial edge contour points is a preset contour point; If so, the target edge contour points are removed from the initial edge contour points, and the remaining edge contour points after removing the target edge contour points from the initial edge contour points are used as the target detection data.

13. A system for constructing a probe position prediction model, characterized in that: The build system includes: A model building module, used to build a three-dimensional body part model corresponding to the sample body part based on the sample detection data corresponding to the sample body part; A virtual probe simulation module, used to simulate virtual probe positioning data and virtual probe scanning data corresponding to the virtual probe according to the three-dimensional data corresponding to the three-dimensional body part model; A sample acquisition module, used for acquiring a two-dimensional structure sample data set corresponding to the sample body part based on the virtual probe positioning data and the virtual probe scanning data; A model training module, used for training the probe position prediction model according to the two-dimensional structure sample data set; The probe position prediction model is used to predict the actual positioning data of the actual detection probe corresponding to the input actual detection data when detecting the actual body part; The actual body part includes a first body part that the actual detection probe can detect, or a second body part that the actual detection probe cannot detect; The sample acquisition module comprises: a three-dimensional intersection point acquisition unit, configured to obtain, for any preset sampling point of the virtual probe in the three-dimensional body part model, initial three-dimensional intersection point data corresponding to the preset sampling point based on the intersection of the virtual probe scanning data corresponding to the preset sampling point and the three-dimensional body part model; An initial data acquisition unit, configured to obtain initial two-dimensional structure data corresponding to the preset sampling point based on the virtual probe positioning data corresponding to the preset sampling point and the initial three-dimensional intersection data; A transformation matrix acquisition unit, used to acquire a random transformation matrix corresponding to the preset sampling point; A sample data construction unit, used for obtaining the two-dimensional structure sample data set based on a plurality of the random transformation matrices and the corresponding initial two-dimensional structure data; The transformation matrix acquisition unit is further used to acquire the random rotation matrix corresponding to the preset sampling point; acquire the target position coordinates corresponding to the preset sampling point; and obtain the random transformation matrix based on the random rotation matrix and the target position coordinates; The model training module is used to train a preset network model based on the two-dimensional structure sample data set to obtain the probe position prediction model; The probe position prediction model includes an input layer, a downsampling layer, an upsampling layer and an output layer; The model training module is used to input the two-dimensional structure sample data set into the input layer for sampling operation, and output the initial actual point set number corresponding to the initial two-dimensional structure data; if the initial actual point set number is greater than the preset point set number, the initial two-dimensional structure data is input into the downsampling layer, and the target two-dimensional structure data is output; if the initial actual point set number is less than the preset point set number, the initial two-dimensional structure data is input into the upsampling layer, and the target two-dimensional structure data is output; wherein the target actual point set number corresponding to the target two-dimensional structure data is equal to the preset point set number; the target two-dimensional structure data is input into the output layer, and the actual positioning data when the actual detection probe detects the actual body part is output.

14. A probe position prediction system, characterized in that: The probe position prediction system comprises a construction system of a probe position prediction model as claimed in claim 13, wherein the probe position prediction system comprises: A detection data acquisition module, used to obtain target detection data corresponding to the actual body part based on the initial detection data collected by the actual detection probe; A probe position prediction module is used to input the target detection data into a probe position prediction model obtained based on the construction system to output actual positioning data when the actual detection probe detects the actual body part.

15. A probe device, characterized in that: The probe device includes a detection probe and a probe position prediction system as claimed in claim 14.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and used to run on the processor, characterized in that: When the processor executes the computer program, the method for constructing a probe position prediction model as described in any one of claims 1 to 9 or the method for predicting a probe position as described in any one of claims 10 to 12 is implemented.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for constructing a probe position prediction model according to any one of claims 1 to 9 or the method for predicting a probe position according to any one of claims 10 to 12 is implemented.

Images