Gastric ultrasound image-assisted scanning method, device and electronic equipment

Abstract

The present invention relates to the field of medical auxiliary technology, and in particular to a method, device, electronic device and storage medium for assisting stomach ultrasound image scanning, wherein the method comprises: obtaining spatial position parameters of an ultrasound probe based on a positioning algorithm; performing segmentation processing on a stomach image using an image segmentation algorithm, and dynamically tracking pixels of a stomach segmentation image region using an image tracking technology to obtain a cross-sectional stomach shape; constructing a three-dimensional stomach model using a three-dimensional reconstruction technology and training a stomach assisting scanning model, using the stomach assisting scanning model to predict an image to be scanned, obtaining labels of various stomach parts, and mapping the labels of various parts to an original three-dimensional image, performing three-dimensional modeling of a patient's stomach based on position information and a deep learning image processing method, so that an ultrasound physician can collect a complete image of the stomach, spatially map the ultrasound image to the three-dimensional model, mark the scanned area, and remind the ultrasound physician to avoid missed detection.

Description

Technical Field

[0001] The present application relates to the field of medical assistance technology, and in particular to a method, device, electronic device and storage medium for assisting gastric ultrasound image scanning. Background Art

[0002] In the field of medical diagnosis, especially in the diagnosis of gastric diseases, ultrasound scanning is widely used due to its non-invasiveness and high efficiency. However, traditional ultrasound scanning technology has certain limitations in ultrasound shooting methods and gastric disease diagnosis. When performing gastric ultrasound, ultrasound physicians often need to rely on their own experience and spatial imagination to confirm the shape and structure of the patient's stomach, and then conduct a detailed scan of the disease-prone structure. The current gastric ultrasound scanning technology has the following main shortcomings:

[0003] (1) Existing ultrasound image scanning operations mostly depend on the subjective judgment of ultrasound physicians based on their experience, and lack an objective and standardized detection method.

[0004] (2) For some gastric diseases (such as gastric cancer), the lesions are not obvious in the early stage, and the stomach is constantly moving or contracting, which makes it difficult for the operator to fully capture all the details of the stomach, resulting in ignoring subtle lesions and easily leading to misdiagnosis or missed diagnosis.

[0005] (3) When clinicians review ultrasound images, they can only obtain the position of the ultrasound probe when the ultrasound physician took the image. They need to use their own experience to determine the specific structure of the image and whether there are any abnormalities. This requires a high level of clinical experience from the doctors, increases the rate of missed diagnoses and misdiagnoses in gastric ultrasound examinations, and affects the popularity of gastric ultrasound examinations.

[0006] In summary, how to design an image-assisted scanning method that can generate the three-dimensional shape of a patient's stomach in real time through scanning and provide accurate guidance is a problem that urgently needs to be solved. Summary of the invention

[0007] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0008] To this end, the first purpose of this application is to propose a gastric ultrasound image-assisted scanning method to solve the problem that the existing technical means are unable to comprehensively capture all details of the stomach, resulting in ignoring subtle lesions, which can easily lead to misdiagnosis and missed diagnosis.

[0009] The second object of the present application is to provide a device.

[0010] The third objective of the present application is to provide an electronic device.

[0011] A fourth objective of the present application is to provide a computer-readable storage medium.

[0012] To achieve the above-mentioned purpose, the first embodiment of the present application proposes a stomach ultrasound image-assisted scanning method, comprising:

[0013] Acquire the acceleration data of the ultrasonic probe, and obtain the spatial position parameters of the ultrasonic probe based on the positioning algorithm;

[0014] The stomach image obtained by ultrasound scanning is segmented using an image segmentation algorithm to obtain a stomach segmentation image;

[0015] Using image tracking technology to dynamically track the pixels of the stomach segmentation image area, obtain the identified stomach wall pixels and calculate the cross-sectional stomach shape;

[0016] Based on the cross-sectional stomach shape and the spatial position parameters, constructing a three-dimensional stomach model using three-dimensional reconstruction technology;

[0017] Training a stomach auxiliary scanning model based on the stomach three-dimensional model to obtain a trained stomach auxiliary scanning model;

[0018] The stomach auxiliary scanning model is used to predict the image to be scanned, obtain labels of various parts of the stomach, and map the labels of various parts to the original three-dimensional image.

[0019] Preferably, the acquiring of the ultrasonic probe acceleration data and obtaining the spatial position parameters of the ultrasonic probe based on a positioning algorithm comprises:

[0020] An inertial measurement unit containing a six-axis accelerometer and a gyroscope is embedded in the ultrasound probe. A complementary filtering algorithm is used to calculate the rotation increment of the gyroscope and the real-time position of the ultrasound probe to obtain the spatial position parameters of the current gastric ultrasound image.

[0021] Preferably, the image segmentation algorithm adopts a U-Net deep neural network, including:

[0022] Annotate the stomach region of the acquired ultrasound image to obtain an annotated image data set;

[0023] Constructing a stomach image segmentation model using a deep neural network, and training the stomach image segmentation model using the annotated image dataset to obtain a trained stomach image segmentation model;

[0024] The trained stomach image segmentation model is used to segment the stomach ultrasound image to obtain a stomach segmentation image.

[0025] Preferably, before labeling the stomach region on the acquired ultrasound image, the process includes:

[0026] The stomach ultrasound image is collected every preset distance from top to bottom until the patient's stomach structure is no longer included in the image, thereby obtaining an image of the patient's stomach structure as a whole.

[0027] Preferably, the step of dynamically tracking the pixels of the stomach segmentation image region by using image tracking technology to obtain the identified stomach wall pixels and calculate the cross-sectional stomach shape comprises:

[0028] According to the sampling frequency of the ultrasound probe, an ultrasound image set is obtained;

[0029] The ultrasound image set is sorted according to timestamps, and the sorted ultrasound images are subjected to scale space extreme value detection, the detected extreme value points are accurately located and screened, and the gradient direction distribution is converted into a vector by a Gaussian weighted descriptor generation method to form a descriptor of a key point;

[0030] Obtain corresponding pixel points in different images based on the descriptor;

[0031] Perform image segmentation on the image set collected at a single location to obtain the gastrointestinal region;

[0032] The weighted average processing is performed on all gastrointestinal areas to obtain the probability that each pixel belongs to the stomach. By judging the threshold, the cross-sectional stomach shape is obtained.

[0033] Preferably, the scale space extreme value detection calculation formula is:

[0034] L(x,y,δ)=(x,y,δ)*(x,y)

[0035]

[0036] Among them, m and n represent the dimensions of the Gaussian template, (x, y) represents the position of the image pixel, δ is the scale space factor, the smaller the value, the less the image is smoothed, and the smaller the corresponding scale, L(x, y, δ) is the scale space.

[0037] Preferably, constructing a three-dimensional stomach model using three-dimensional reconstruction technology based on the cross-sectional stomach shape and the spatial position parameters includes:

[0038] Acquiring a stomach contour image based on the stomach shape of the section;

[0039] A Cartesian space coordinate system is constructed based on the stomach contour image, and each stomach contour image is positioned in the coordinate system according to its spatial position parameters. The blank space between each sampled image is filled with interpolation method, and the position information of four adjacent stomach contour images is fitted and calculated to obtain a three-dimensional stomach model.

[0040] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a stomach ultrasound image-assisted scanning device, comprising:

[0041] The spatial position acquisition module acquires the acceleration data of the ultrasonic probe and obtains the spatial position parameters of the ultrasonic probe based on the positioning algorithm;

[0042] An image segmentation module uses an image segmentation algorithm to segment the stomach image obtained by ultrasound scanning to obtain a stomach segmentation image;

[0043] A section calculation module, which uses image tracking technology to dynamically track the pixels of the stomach segmentation image area, obtains the identified stomach wall pixels and calculates the section stomach shape;

[0044] A three-dimensional model building module, based on the cross-sectional stomach shape and the spatial position parameters, uses three-dimensional reconstruction technology to build a three-dimensional stomach model;

[0045] A training module, which trains the stomach auxiliary scanning model based on the stomach three-dimensional model to obtain a trained stomach auxiliary scanning model;

[0046] The prediction module uses the stomach auxiliary scanning model to predict the image to be scanned, obtains the labels of various parts of the stomach, and maps the labels of various parts to the original three-dimensional image.

[0047] To achieve the above-mentioned purpose, the third aspect of the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0048] The memory stores computer-executable instructions;

[0049] The processor executes the computer-executable instructions stored in the memory to implement any of the above methods.

[0050] To achieve the above-mentioned purpose, the fourth aspect embodiment of the present application proposes a computer-readable storage medium, including computer-executable instructions stored in the computer-readable storage medium, and the computer-executable instructions are used to implement any of the methods described above when executed by a processor.

[0051] The present application provides a method for assisting gastric ultrasound image scanning, which combines an inertial measurement unit (IMU) with an ultrasound probe, uses a position calculation algorithm to measure the real-time position of the probe, obtains the position information of each ultrasound image, and performs three-dimensional modeling of the patient's stomach based on the position information and a deep learning image processing method, so that the ultrasound physician can collect a complete image of the stomach, track and locate the moving stomach through the IMU and image tracking algorithm, perform spatial position mapping of the ultrasound image and the three-dimensional model, mark the scanned area, and remind the ultrasound physician to avoid missed detection.

[0052] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0054] Figure 1 This is a flow chart of a first specific embodiment of a stomach ultrasound image-assisted scanning method provided by the present invention;

[0055] Figure 2 It is a technical flow chart of the present invention;

[0056] Figure 3 Label the resulting map for the stomach region;

[0057] Figure 4 This is a three-dimensional reconstruction of the stomach;

[0058] Figure 5 Provide a three-dimensional annotation effect diagram of the stomach structure;

[0059] Figure 6 A structural block diagram of a gastric ultrasound image-assisted scanning device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The core of the present invention is to provide a gastric ultrasound image-assisted scanning method, device, electronic device and storage medium, which uses a deep learning algorithm to accurately identify various gastric structures on a reconstructed three-dimensional gastric model. The deep learning algorithm can also detect abnormal points in real time and guide doctors to perform precise operations on suspicious areas to screen for abnormal lesions.

[0061] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0062] Please refer to Figure 1 , Figure 1 This is a flow chart of a first specific embodiment of a stomach ultrasound image-assisted scanning method provided by the present invention; the specific operation steps are as follows:

[0063] Step S101: Acquire the acceleration data of the ultrasonic probe, and obtain the spatial position parameters of the ultrasonic probe based on the positioning algorithm;

[0064] An inertial measurement unit containing a six-axis accelerometer and a gyroscope is embedded in the ultrasound probe. A complementary filtering algorithm is used to calculate the rotation increment of the gyroscope and the real-time position of the ultrasound probe to obtain the spatial position parameters of the current gastric ultrasound image.

[0065] The stomach ultrasound image is collected every preset distance from top to bottom until the patient's stomach structure is no longer included in the image, thereby obtaining an image of the patient's stomach structure as a whole.

[0066] Step S102: using an image segmentation algorithm to segment the stomach image acquired by ultrasound scanning to obtain a stomach segmentation image;

[0067] The image segmentation algorithm uses a U-Net deep neural network, including:

[0068] Annotate the stomach region of the acquired ultrasound image to obtain an annotated image data set;

[0069] Constructing a stomach image segmentation model using a deep neural network, and training the stomach image segmentation model using the annotated image dataset to obtain a trained stomach image segmentation model;

[0070] The trained stomach image segmentation model is used to segment the stomach ultrasound image to obtain a stomach segmentation image.

[0071] Step S103: dynamically tracking the pixels of the stomach segmentation image region using image tracking technology, obtaining the identified stomach wall pixels and calculating the cross-sectional stomach shape;

[0072] According to the sampling frequency of the ultrasound probe, an ultrasound image set is obtained;

[0073] The ultrasound image set is sorted according to timestamps, and the sorted ultrasound images are subjected to scale space extreme value detection, the detected extreme value points are accurately located and screened, and the gradient direction distribution is converted into a vector by a Gaussian weighted descriptor generation method to form a descriptor of a key point;

[0074] Obtain corresponding pixel points in different images based on the descriptor;

[0075] Perform image segmentation on the image set collected at a single location to obtain the gastrointestinal region;

[0076] The weighted average processing is performed on all gastrointestinal areas to obtain the probability that each pixel belongs to the stomach. By judging the threshold, the cross-sectional stomach shape is obtained.

[0077] The scale space extreme value detection calculation formula is:

[0078] L(x,y,δ)=(x,y,δ)*(x,y)

[0079]

[0080] Among them, m and n represent the dimensions of the Gaussian template, (x, y) represents the position of the image pixel, δ is the scale space factor, the smaller the value, the less the image is smoothed, and the smaller the corresponding scale, L(x, y, δ) is the scale space.

[0081] Step S104: constructing a three-dimensional model of the stomach using three-dimensional reconstruction technology based on the cross-sectional stomach shape and the spatial position parameters;

[0082] Acquiring a stomach contour image based on the stomach shape of the section;

[0083] A Cartesian space coordinate system is constructed based on the stomach contour image, and each stomach contour image is positioned in the coordinate system according to its spatial position parameters. The blank space between each sampled image is filled with interpolation method, and the position information of four adjacent stomach contour images is fitted and calculated to obtain a three-dimensional stomach model.

[0084] Step S105: training the stomach auxiliary scanning model based on the stomach three-dimensional model to obtain a trained stomach auxiliary scanning model;

[0085] Step S106: using the stomach auxiliary scanning model to predict the image to be scanned, obtaining labels of various parts of the stomach, and mapping the labels of various parts to the original three-dimensional image.

[0086] This embodiment provides a method for assisting the scanning of gastric ultrasound images, which is calibrated by accelerometers and gyroscopes to calculate the relative position relationship between the patient's stomach and the ultrasound probe, providing basic data for subsequent image processing. Then, the image processing algorithm is used to accurately segment each frame of the stomach image obtained by ultrasound scanning to accurately identify the stomach area. In addition, the video image tracking technology is applied to dynamically track the pixels of the segmented stomach area to ensure that the identified stomach wall pixels are captured in real time during the peristalsis or contraction of the stomach, and the shape of the stomach in this section is calculated. Subsequently, the three-dimensional reconstruction of the stomach shape is performed using the saved stomach shape image and the relative position data of the probe to form an intuitive three-dimensional model. Then, a deep learning method is used to identify the stomach structure to improve the accuracy and comprehensiveness of the identification. Finally, combined with the ultrasound scanning experience of senior doctors, a unified ultrasound scanning operation standard is formed, and the operation steps are marked with text or images on the three-dimensional model, and operation correction opinions are provided according to the real-time position of the ultrasound probe.

[0087] Based on the above embodiment, this embodiment describes the stomach ultrasound image-assisted scanning method. Figure 2As shown, the details are as follows:

[0088] This embodiment requires some auxiliary equipment including: a six-axis accelerometer, an inertial measurement unit of a gyroscope, and an ultrasonic probe.

[0089] In order to perform three-dimensional modeling of the patient's stomach shape through ultrasound images, an inertial measurement unit (IMU) containing a six-axis accelerometer and a gyroscope is embedded in the ultrasound probe to calculate the real-time position of the ultrasound probe, thereby obtaining the spatial position parameters of the current stomach ultrasound image.

[0090] A gyroscope is a sensor that uses the law of conservation of angular momentum to measure the angular velocity of an object. Within a time period dt, the gyroscope can measure the real-time angular velocity of the ultrasound probe. The measured data are GyroX, GyroY, and GyroZ, which represent the real-time angular velocity of the ultrasound probe on the X-axis, Y-axis, and Z-axis, respectively. The accelerometer measures the real-time acceleration of the X-axis, Y-axis, and Z-axis of the ultrasound probe. The measured data are ax, ay, and az, which represent the acceleration of the X-axis, Y-axis, and Z-axis, respectively.

[0091] When the ultrasound physician begins the operation, he needs to press the calibration button one centimeter above the patient's navel to reset the IMU to the initial posture orientation 0. The motion posture of the IMU is represented by quaternion, and the initial posture is orientation0 = [1,0,0,0]. The motion trajectory is represented by a three-dimensional space vector, and the initial position is pos0 = [0,0,0].

[0092] The quaternion is a mathematical tool for representing rotation in three-dimensional space, which consists of a real part and three imaginary parts, usually expressed as q=w+xi+yj+zk, where w, x, y, z are real numbers.

[0093] After the initial posture is set, the IMU will measure the ultrasound probe acceleration data and gyroscope angular velocity data in real time according to the ultrasound physician's operation. Because the gyroscope is sensitive to short-term dynamic changes, but long-term integration will produce error accumulation, and the accelerometer has long-term stability, but is disturbed by motion acceleration under short-term dynamics, this method adopts the existing complementary filtering algorithm. When calculating the rotation increment △A, it is necessary to combine the gyroscope rotation increment △A with short-term accuracy. gyro (△A gyro =[△A gyrox ,△A gyroy ,△A gyroz ]) and the accelerometer rotation increment △A with long-term stability acc (△A acc =[△A accx ,△a accy ,△A accz ]) Comprehensively estimate the real-time spatial position of the ultrasound probe.

[0094] For gyroscope data, the rotation increment can be calculated directly from the angular velocity of each axis. The formula is as follows:

[0095]

[0096] For the accelerometer data, calculate the pitch angle △A accx and roll angle △A accy The pitch angle is the rotation angle of the ultrasound probe around the X-axis, which is used to determine the degree of its forward or backward tilt. The calculation formula is:

[0097]

[0098] The roll angle is the rotation angle of the ultrasound probe around the Y axis, which is used to determine the degree of inclination of the object to the right or left. The calculation formula is:

[0099] △A accy =arctan2(a y ,a z )*dt

[0100] In order to obtain a more accurate attitude angle, the complementary filtering formula for fusing the two is:

[0101]

[0102] Among them, A t+1 x , A t+1 y , A t+1 z Represents the attitude angles of the ultrasound probe around the X-axis, Y-axis, and Z-axis at the next moment; A t x , A t y , A t z It represents the attitude angle of the ultrasonic probe at the current moment; α is the complementary filter weight coefficient, and the recommended value is 0.8; for the data received by the IMU sensor, the attitude angle of the ultrasonic probe at each timestamp can be obtained.

[0103] After that, according to the current posture angle of the ultrasonic probe, the velocity of the current ultrasonic probe along each coordinate axis can be calculated by integration. The initial velocity v0 is 0, and the velocity v of the current ultrasonic probe can be expressed as:

[0104] v t =v t-1 +[ax,ay,az]*dt

[0105] Among them, v t-1 represents the velocity at the previous moment, vt Indicates the speed at the current moment.

[0106] The velocity v of the ultrasonic probe is integrated to obtain the specific position pos of the ultrasonic probe. pos is a three-dimensional vector containing the real-time position information of the ultrasonic probe. The format is pos=[p x ,p y ,p z ], the initial position pos coordinate is [0,0,0], and the velocity integral formula is:

[0107] POS t =pos t-1 +v t *dt

[0108] Among them, pos t-1 Indicates the position at the last moment, pos t Indicates the current position.

[0109] In order to acquire the overall structural image of the patient's stomach, it is necessary to acquire stomach ultrasound images at intervals from top to bottom. The recommended vertical distance interval is △p z 1 cm, until the image does not contain the patient's stomach structure; the collected images are P i (i=1-n);

[0110] The three-dimensional modeling of the stomach is established based on the shape slices of the stomach and the corresponding spatial position. Therefore, the ultrasound image scanned by the doctor needs to be segmented using an image segmentation algorithm to obtain the corresponding stomach shape slices. The image segmentation algorithm uses the U-Net deep neural network, and the algorithm model is divided into a training phase and an application phase.

[0111] Algorithm training phase: First, use the annotation software to annotate the stomach area of ​​the collected ultrasound images under the guidance of professional physicians. The results after the annotated area is exported are as follows: Figure 3 shown.

[0112] The exported regional image and the original ultrasound image are used as input and trained using the U-Net deep neural network architecture;

[0113] The U-Net architecture consists of three parts: encoder, decoder, and skip connection.

[0114] Algorithm input input = [w i ,h i ,c i ],w i is the width of the image, h i is the height of the image, c i is the number of channels of the image. The output of the algorithm is [wo ,h o ,c o ],w o =w i ,h o =h i .c o is 2, indicating the probability of the stomach region and other regions of no interest in the input image.

[0115] The encoder extracts the input features through a convolution kernel with 64 channels and a size of 3*3. After performing two convolution operations and using the ReLu activation function, the output features are reduced in dimension through a 2*2 maximum pooling layer. After that, the obtained output features are sequentially passed through a convolution operation with 128 channels, an activation function, a convolution operation with 128 channels, an activation function, and a pooling layer. This is repeated multiple times, with the number of channels doubling each time, until a feature value output with 1024 channels is obtained.

[0116] The decoder Deccoder uses a 2*2 upsampling convolution to increase the dimension of the encoder output, while the number of channels remains unchanged. After that, the result of the encoder at the same layer is fused with the upsampling result of the decoder as the input of the decoder layer. Then, a convolution kernel with a channel number of 512 and a size of 3*3 is used to extract features. After two convolution operations, the output features are increased in dimension through upsampling convolution. After that, the upsampling result is sequentially skipped with the output of the encoder at the same layer, convolution operation with a channel number of 256, activation function, convolution operation with a channel number of 256, activation function, and upsampling convolution. Repeat this process many times until the number of channels is 64, then no upsampling convolution is performed, but a convolution operation with a channel number of 2 is performed to obtain the final output ouput. The two channel numbers of ouput represent the probability that each pixel in the target image belongs to the stomach and the probability of other areas of no interest.

[0117] The output result is judged by threshold, and the pixels in the image are processed. The pixels with probability greater than the threshold are marked as white, otherwise they are marked as black. The threshold is generally set to 0.7.

[0118] Algorithm application stage: The ultrasound image scanned by the ultrasound probe is used as input to output the corresponding stomach shape slice.

[0119] The stomach area obtained by only segmenting the stomach ultrasound image is not accurate enough, so image tracking technology is needed to improve the accuracy of stomach area recognition. When ultrasound physicians perform stomach ultrasound detection, the ultrasound image will continue to change due to the continuous peristalsis of the stomach and intestines. In order to obtain reference stomach shape slices at each vertical position for three-dimensional modeling, when physicians use ultrasound probes, they need to scan a clear image at each vertical position and stay until there are no motion artifacts in the image. The recommended time is 1 second. This is to track each pixel in the image and weight the segmentation results of multiple stomach shape slices at the same position to improve the accuracy of ultrasound image segmentation. The specific process is as follows:

[0120] Image matching: According to the sampling frequency of the ultrasound probe, a series of ultrasound images can be obtained, denoted as N, and these images (img1 to img N ) According to the following logic processing, the SIFT scale-invariant feature transformation algorithm is used to match the motion process of each pixel in the image, as follows:

[0121] Perform scale space extrema detection on the image. The scale space L(x,y,δ) of an image is defined as the convolution of a Gaussian function G(x,y,δ) with a varying scale and the original image I(x,y), that is:

[0122] L(x,y,δ)=(x,y,δ)*(x,y)

[0123]

[0124] Among them, m and n represent the dimensions of the Gaussian template, (x, y) represents the position of the image pixel, and δ is the scale space factor. The smaller the value, the less the image is smoothed and the smaller the corresponding scale.

[0125] A scale space is constructed at different scales. At each pixel position in the scale space, the extreme point is detected by comparing its value with that of the adjacent pixels and the pixels at the adjacent scales. Then the key point is located, and the detected extreme points are accurately located and screened. In order to make the key point rotationally invariant, the algorithm calculates the gradient magnitude and direction in the neighborhood around the key point, and then finds the main direction on the gradient direction histogram. The direction with the largest gradient magnitude is taken as the main direction, and the direction greater than or equal to 80% of the main direction magnitude is taken as the auxiliary direction. According to the number of auxiliary directions, the key point is regarded as multiple key points, and the corresponding direction information is assigned to each key point. After finding the position, scale, and direction of the key point, its invariance in movement, scaling, and rotation can be ensured. In addition, a descriptor vector needs to be established for the key point so that it can remain invariant under different light and viewing angles and can be distinguished from other key points. The 16*16 pixel area around the key point is divided into 16 4*4 sub-areas. For each sub-area, the gradient magnitude and direction of each pixel in it are calculated and added to the histogram of the 8 directions of the sub-area. Finally, a 128-dimensional SIFI descriptor is obtained, corresponding to 8 directions of 16 sub-regions, to achieve invariance to illumination changes.

[0126] SIFT uses a Gaussian weighted descriptor generation method to convert the gradient direction distribution into a vector to form a descriptor of the key point. By comparing the descriptors of the key points, feature matching can be performed to find the corresponding pixels in different images.

[0127] Weighted calculation of segmentation results: The image set collected at a single location is sorted according to the timestamp, and these images (img1~img N ) is processed according to the following logic:

[0128] Perform image segmentation on img1 to obtain the gastrointestinal region ROIS1.

[0129] For img i ,i∈[2,N]. According to img i-1 Image segmentation results ROIS i-1 And pixel tracking results ROIC i-11 ~ROIC i-1(i-1)-1 , respectively, perform pixel tracking to obtain ROIC i1 ~ROIC ii-1 , for img i The result of image segmentation is ROIS i .

[0130] After processing these N images, the number of gastrointestinal regions obtained is N(N+1) / 2. These regions are weighted averaged to obtain the probability that each pixel belongs to the stomach. By comparing with the threshold, the stomach shape slice of the patient at this position is finally obtained.

[0131] Using the stomach shape slices and the spatial position parameters of the probe, a three-dimensional stomach model is constructed through three-dimensional reconstruction technology. The specific steps are as follows. This three-dimensional model can intuitively display the spatial structure of the stomach and provide assistance to doctors in ultrasound examination operations.

[0132] The stomach shape slices in the stomach shape slices are extracted to generate a stomach contour image. A Cartesian space coordinate system is established with 1 cm above the patient's navel as the origin, the vertical abdomen outward as the X-axis, the vertical ground as the Z-axis, and the vertical to the XOZ plane as the Y-axis; each stomach contour image is positioned in the space coordinate system according to its spatial position parameters. Because each sampled image is separated by a certain height when the doctor collects ultrasound images, there will be blank spaces in the middle. In order to generate a continuous three-dimensional stomach model, interpolation method is required to fill it. When filling the blank space, the position information of the four adjacent stomach contour images is fitted and calculated.

[0133] For the blank area between every two stomach contour images Pi and Pi+1, we additionally take a stomach contour image Pi-1 above Pi and a stomach contour image Pi+2 below Pi+1, and simultaneously take four points located in the same vertical plane on the four stomach contour images, and use the Lagrange interpolation method to perform cubic function fitting. The formula is as follows:

[0134]

[0135] Among them, each basis function L i () is defined as:

[0136]

[0137] Then, Lagrange interpolation method is used to fit the blank space between each contour image to obtain a complete 3D model. Figure 4 shown.

[0138] Structural recognition: Use deep learning methods to identify the structures in the 3D reconstructed stomach model, including key areas such as the gastric fundus, gastric antrum, and pylorus, to improve the accuracy of identifying these areas. Figure 5 shown.

[0139] The 3D U-Net network is used to identify key parts of the three-dimensional stomach model. The 3D U-Net process is the same as the U-Net process described in step 3 image segmentation. The difference between the two is the different dimensions of the input data. The 3D U-Net architecture uses three-dimensional convolution, three-dimensional maximum pooling and three-dimensional upsampling, which enables the model to effectively process and analyze three-dimensional spatial information. In the volume segmentation task, it can better understand and segment the three-dimensional structure.

[0140] Then the data set is divided into training set, validation set and test set in a ratio of 7:2:1. The 3D stomach images in the training set are segmented by sliding, and the segmented data is input into the 3D U-Net network for training to obtain the optimal network model. The optimal model is used for prediction to obtain the labels of various parts of the stomach, and finally the segmented label data is mapped back to the original 3D image, the structure in the 3D stomach image is annotated in real time, and finally the segmented label data is mapped back to the original 3D image. Find that to provide real-time auxiliary prompts during the ultrasound physician's operation.

[0141] The embodiment of the present invention provides a gastric ultrasound image assisted scanning method, which assists ultrasound doctors in gastric ultrasound scanning through image processing, deep learning and three-dimensional reconstruction technology, combines an inertial measurement unit (IMU) with an ultrasound probe, uses a position calculation algorithm to measure the real-time position of the probe, obtains the position information of each ultrasound image, and performs three-dimensional modeling of the patient's stomach based on the position information and the deep learning image processing method, so that ultrasound doctors can collect complete images of the stomach, and it is also convenient for clinicians to obtain three-dimensional information of ultrasound images during diagnosis, reducing the misdiagnosis and missed diagnosis rate. Combining three-dimensional modeling technology with image processing technology, a unified and systematic ultrasound examination process can be proposed for different stomach shapes, that is, first scan the entire stomach from top to bottom, generate a three-dimensional model of the stomach shape, then perform structural recognition on the three-dimensional model, and finally guide the ultrasound doctor to scan the stomach structure one by one. When the ultrasound doctor scans the stomach structure, the moving stomach is tracked and located through the IMU and image tracking algorithm, the ultrasound image and the three-dimensional model are spatially mapped, and the scanned area is marked to remind the ultrasound doctor to avoid missed detection. When clinicians view the patient's ultrasound image, the spatial position of the stomach when the image is collected is marked through a three-dimensional model, so that clinicians can immediately obtain the specific position of the image in the stomach, and the collected stomach ultrasound image is spatially marked in the three-dimensional model. Clinicians click on the spatial mark in the three-dimensional image to view the ultrasound image of the corresponding part, which greatly saves the clinicians' time.

[0142] Please refer to Figure 6 , Figure 6 A structural block diagram of a stomach ultrasound image auxiliary scanning device provided by an embodiment of the present invention; the specific device may include:

[0143] The spatial position acquisition module 100 acquires the acceleration data of the ultrasonic probe and obtains the spatial position parameters of the ultrasonic probe based on the positioning algorithm;

[0144] An image segmentation module 200 uses an image segmentation algorithm to segment the stomach image obtained by ultrasound scanning to obtain a stomach segmentation image;

[0145] The section calculation module 300 uses image tracking technology to dynamically track the pixels of the stomach segmentation image area, obtain the identified stomach wall pixels and calculate the section stomach shape;

[0146] A three-dimensional model building module 400, based on the cross-sectional stomach shape and the spatial position parameters, uses a three-dimensional reconstruction technology to build a three-dimensional stomach model;

[0147] A training module 500 is used to train a stomach auxiliary scanning model based on the stomach three-dimensional model to obtain a trained stomach auxiliary scanning model;

[0148] The prediction module 600 uses the stomach auxiliary scanning model to predict the image to be scanned, obtains the labels of various parts of the stomach, and maps the labels of various parts to the original three-dimensional image.

[0149] A gastric ultrasound image assisted scanning device of the present embodiment is used to implement the aforementioned gastric ultrasound image assisted scanning method. Therefore, the specific implementation method of a gastric ultrasound image assisted scanning device can be seen in the embodiment part of a gastric ultrasound image assisted scanning method in the previous text. For example, the spatial position acquisition module 100, the image segmentation module 200, the section calculation module 300, the three-dimensional model construction module 400, the training module 500, and the prediction module 600 are respectively used to implement steps S101, S102, S103, S104, S105, and S106 in the aforementioned gastric ultrasound image assisted scanning method. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part, which will not be repeated here.

[0150] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0151] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0152] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0153] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0154] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0155] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0156] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0157] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0158] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0159] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0160] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0161] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0162] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0163] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A stomach ultrasound image-assisted scanning method, characterized in that: include: Acquire the acceleration data of the ultrasound probe, and obtain the spatial position parameters of the ultrasound probe based on the positioning algorithm, wherein an inertial measurement unit including a six-axis accelerometer and a gyroscope is embedded in the ultrasound probe, and a complementary filtering algorithm is used to calculate the rotation increment of the gyroscope, calculate the real-time position of the ultrasound probe, and obtain the spatial position parameters of the current stomach ultrasound image; Using an image segmentation algorithm to segment a stomach image acquired by ultrasound scanning to obtain a stomach segmentation image, wherein the collected ultrasound image is annotated with a stomach region to obtain an annotated image data set, a stomach image segmentation model is constructed using a deep neural network, the annotated image data set is used to train the stomach image segmentation model to obtain a trained stomach image segmentation model, and the trained stomach image segmentation model is used to segment the stomach ultrasound image to obtain a stomach segmentation image; The pixels of the stomach segmentation image area are dynamically tracked by using image tracking technology to obtain the identified stomach wall pixels and calculate the cross-sectional stomach shape, wherein an ultrasound image set is obtained according to the sampling frequency of the ultrasound probe, the ultrasound image set is sorted according to the timestamp, and the sorted ultrasound images are subjected to scale space extreme value detection, the detected extreme value points are accurately located and screened, the gradient direction distribution is converted into a vector by a Gaussian weighted descriptor generation method to form a descriptor of the key point, the corresponding pixel points in different images are obtained based on the descriptor, the image set collected at a single position is subjected to image segmentation to obtain the gastrointestinal area, all the gastrointestinal areas are subjected to weighted average processing to obtain the probability that each pixel point belongs to the stomach, and the cross-sectional stomach shape is obtained by judging the threshold; Based on the cross-sectional stomach shape and the spatial position parameters, constructing a three-dimensional stomach model using three-dimensional reconstruction technology; Training a stomach auxiliary scanning model based on the stomach three-dimensional model to obtain a trained stomach auxiliary scanning model; The stomach auxiliary scanning model is used to predict the image to be scanned, obtain labels of various parts of the stomach, and map the labels of various parts to the original three-dimensional image.

2. The method for stomach ultrasound image-assisted scanning according to claim 1, characterized in that: Before labeling the stomach region of the acquired ultrasound image, the method includes: The stomach ultrasound image is collected every preset distance from top to bottom until the patient's stomach structure is no longer included in the image, thereby obtaining an image of the patient's stomach's overall structure.

3. The stomach ultrasound image-assisted scanning method according to claim 1, characterized in that: The scale space extreme value detection calculation formula is: L(x,y,δ)=G(x,y,δ)*I(x,y) Among them, m and n represent the dimensions of the Gaussian template, (x, y) represents the position of the image pixel, δ is the scale space factor, the smaller the value, the less the image is smoothed, and the smaller the corresponding scale, L(x, y, δ) is the scale space.

4. The stomach ultrasound image-assisted scanning method according to claim 1, characterized in that: The constructing of a three-dimensional stomach model by using three-dimensional reconstruction technology based on the cross-sectional stomach shape and the spatial position parameters includes: Acquiring a stomach contour image based on the stomach shape of the section; A Cartesian space coordinate system is constructed based on the stomach contour image, and each stomach contour image is positioned in the coordinate system according to its spatial position parameters. The blank space between each sampled image is filled with interpolation method, and the position information of four adjacent stomach contour images is fitted and calculated to obtain a three-dimensional stomach model.

5. A stomach ultrasound image-assisted scanning device, characterized in that: include: The spatial position acquisition module acquires the acceleration data of the ultrasound probe and obtains the spatial position parameters of the ultrasound probe based on the positioning algorithm. The inertial measurement unit including the six-axis accelerometer and the gyroscope is embedded in the ultrasound probe, and the complementary filtering algorithm is used to calculate the rotation increment of the gyroscope, calculate the real-time position of the ultrasound probe, and obtain the spatial position parameters of the current stomach ultrasound image. An image segmentation module, which uses an image segmentation algorithm to segment the stomach image acquired by ultrasound scanning to obtain a stomach segmentation image, wherein the collected ultrasound image is annotated with a stomach region to obtain an annotated image data set, a stomach image segmentation model is constructed using a deep neural network, the stomach image segmentation model is trained using the annotated image data set to obtain a trained stomach image segmentation model, and the stomach ultrasound image is segmented using the trained stomach image segmentation model to obtain a stomach segmentation image; A section calculation module, which uses image tracking technology to dynamically track the pixels of the stomach segmentation image area, obtains the identified stomach wall pixels and calculates the section stomach shape, wherein an ultrasound image set is obtained according to the sampling frequency of the ultrasound probe, the ultrasound image set is sorted according to the timestamp, and the sorted ultrasound images are subjected to scale space extreme value detection, the detected extreme value points are accurately located and screened, and the gradient direction distribution is converted into a vector by a Gaussian weighted descriptor generation method to form a descriptor of the key point, and the corresponding pixel points in different images are obtained based on the descriptor, and the image set collected at a single position is subjected to image segmentation to obtain the gastrointestinal area, and all gastrointestinal areas are subjected to weighted average processing to obtain the probability that each pixel point belongs to the stomach, and the section stomach shape is obtained by judging the threshold; A three-dimensional model building module, based on the cross-sectional stomach shape and the spatial position parameters, uses three-dimensional reconstruction technology to build a three-dimensional stomach model; A training module, which trains the stomach auxiliary scanning model based on the stomach three-dimensional model to obtain a trained stomach auxiliary scanning model; The prediction module uses the stomach auxiliary scanning model to predict the image to be scanned, obtains the labels of various parts of the stomach, and maps the labels of various parts to the original three-dimensional image.

6. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 4 when executed by a processor.

Images