Motion artifact correction method and device for prone milk scanning

Abstract

The invention relates to the technical field of artifact correction, and discloses a motion artifact correction method and device for prone breast scanning, and the method comprises the steps: obtaining an image sequence data set of breast tissues; inputting the image sequence data set into a pre-established deep learning model to obtain a motion field between adjacent images; correcting the image sequence data set based on the motion field to obtain a corrected image sequence data set; the corrected image sequence data set is used for constructing a mammary gland three-dimensional image. According to the invention, artifact correction is carried out based on a motion field method, the displacement vector between the images can be directly estimated, and according to the displacement vector, image errors caused by motion of a patient or equipment can be more accurately corrected, so that the artifact correction precision is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artifact correction, in particular to a motion artifact correction method and device for prone breast scanning. BACKGROUND

[0002] Breast cancer is one of the most common malignant tumors in the world, and early screening is crucial to improve the survival rate of patients. Breast ultrasound imaging, due to its non-radiation, non-invasiveness, and real-time nature, has become an important tool for breast disease screening and diagnosis, especially for patients with high breast density.

[0003] However, during the prone breast ultrasound scanning process, patient breathing, heart beating, muscle tension, etc. can all affect the position of the breast. For example, during the patient's breathing process, the movement of the chest may cause the breast to shift up and down or left and right; heart beating can cause slight vibration of the breast tissue, especially in the breast tissue close to the chest wall; unconscious muscle tension, body position adjustment, etc. can also have a slight impact on the position of the breast. These slight movements can cause slight misalignment of adjacent frame images during breast ultrasound scanning, and in the process of three-dimensional reconstruction, may produce artifacts such as blurring, misalignment, stretching, and repetition, affecting the accuracy of the final image and reducing the accuracy of the doctor's diagnosis of the lesion.

[0004] Current artifact correction methods are not sensitive enough to respond to subtle movements. For example, image processing-based methods for artifact correction rely on global or local image features, and when dealing with small motion displacements, it is difficult to accurately capture these subtle movements, resulting in limitations in artifact correction. SUMMARY

[0005] Therefore, the present application provides a motion artifact correction method and device for prone breast scanning to solve the problem that the motion artifact correction method in the prior art is difficult to capture subtle movements, resulting in limited artifact correction accuracy.

[0006] In a first aspect, the present application provides a motion artifact correction method for prone breast scanning, which comprises:

[0007] obtaining an image sequence data set of breast tissue;

[0008] inputting the image sequence data set into a pre-established deep learning model to obtain a motion field between adjacent images; the deep learning model comprises a segmentation branch and a motion residual branch; the segmentation branch is used for tissue segmentation of consecutive two images to obtain tissue segmentation data; the motion detection branch is used for calculating the motion change parameter of each pixel point in the consecutive two images; and the generation of the motion field is guided based on the tissue segmentation data and the motion change parameter;

[0009] Based on the motion field, the image sequence data set is corrected to obtain a corrected image sequence data set, and the corrected image sequence data set is used to construct a three-dimensional image of the breast.

[0010] The artifact correction method based on the motion field can directly estimate the displacement vector between images, and according to the displacement vector, the image error caused by patient or device motion can be more accurately corrected. Compared with the traditional image registration method, the motion field can capture more local motion characteristics and provide more detailed motion compensation. Moreover, the motion field correction can directly remove or reduce the motion artifacts from the image without relying on the alignment of the overall image. Especially in ultrasound images, the motion field correction can better handle dynamic changes such as periodic motion caused by breathing or heartbeat, thereby improving the accuracy of artifact correction.

[0011] In an optional embodiment, after obtaining the corrected image sequence data set, the method further comprises:

[0012] Statistically correcting the motion vector data of all motion fields in the corrected image sequence data set;

[0013] Converting the motion vector data to the frequency domain to obtain frequency domain motion data;

[0014] Based on the frequency domain motion data, determining an abnormal motion pattern;

[0015] Determining an abnormal motion field corresponding to the abnormal motion pattern;

[0016] Correcting the abnormal image corresponding to the abnormal motion field to obtain a target image sequence data set;

[0017] Based on the target image sequence data set, constructing a three-dimensional image of the breast.

[0018] In this embodiment, based on the corrected image sequence data, the images deviating from the main motion pattern are corrected again through the identification of the motion pattern, thereby further improving the accuracy of artifact correction.

[0019] In an optional embodiment, based on the frequency domain motion data, the abnormal motion pattern is determined, comprising:

[0020] Inputting the frequency domain motion data into a pre-established motion pattern recognition model to determine a main motion pattern;

[0021] Based on the main motion pattern, the abnormal motion pattern is determined.

[0022] In this embodiment, the frequency domain data and the deep learning model are used to identify the main motion pattern, which can effectively improve the accuracy of main motion pattern identification, thereby facilitating the correction accuracy of abnormal images in the abnormal motion pattern.

[0023] In an optional implementation, the abnormal image corresponding to the abnormal motion field is corrected, including:

[0024] determining a normal motion field corresponding to a normal motion pattern adjacent to the abnormal motion pattern;

[0025] correcting the abnormal image based on the normal motion field and the abnormal motion field.

[0026] In this embodiment, the motion field of the image sequence can be ensured to be smoother and more consistent through this correction method, thereby improving the accuracy of artifact correction.

[0027] In an optional implementation, the abnormal image is corrected based on the normal motion field and the abnormal motion field, including:

[0028] determining an average value of the normal motion field and the abnormal motion field;

[0029] calculating displacement correction data based on the abnormal motion field and the average value;

[0030] correcting the abnormal image based on the displacement correction data.

[0031] In this embodiment, by correcting the motion field in the image sequence, smooth transition between images can be ensured, and unnecessary motion deviation can be eliminated. The change of the motion field between adjacent frames is ensured to conform to the expected main motion pattern, avoiding unnatural jumps or mutations, thereby effectively improving the accuracy of three-dimensional reconstruction and providing more reliable image data support for doctors.

[0032] In an optional implementation, the motion field between adjacent images in the image sequence dataset is determined, including:

[0033] inputting the image sequence dataset into a pre-established deep learning model to obtain the motion field between adjacent images; the deep learning model includes a processing module for removing image noise, non-breast tissue regions, and rotation centerline opposite regions.

[0034] In this embodiment, the method not only improves the calculation efficiency, but also accurately captures the subtle motion changes between images, and can maintain high precision even in complex dynamic scenes.

[0035] In an optional implementation, before obtaining the motion field between adjacent images, it further includes:

[0036] obtaining relative motion information of the ultrasound probe between consecutive frames;

[0037] establishing a rigid motion geometric transformation model based on the relative motion information;

[0038] Based on a rigid motion geometric transformation model, each frame of image in the image sequence dataset is geometrically transformed to obtain a motion-corrected image sequence dataset;

[0039] The motion-corrected image sequence dataset is used as the image sequence dataset.

[0040] In an optional implementation, the image sequence dataset is input into a pre-established deep learning model to obtain a motion field between adjacent images, including:

[0041] A preliminary motion field between adjacent images determined by the deep learning model in a learning process is obtained.

[0042] A corresponding heartbeat time sequence when the image sequence dataset is collected is obtained.

[0043] Based on the heartbeat time sequence, breast tissue periodic motion information is determined.

[0044] Based on the breast tissue periodic motion information, consistency verification is performed on the preliminary motion field.

[0045] In a case where the consistency verification is passed, a corresponding motion field is output.

[0046] In the implementation, the physiological reasonableness and the physical consistency of the estimated motion field of the deep learning model can be effectively improved.

[0047] In a second aspect, the present application provides a motion artifact correction device for prone breast scanning, which comprises:

[0048] An acquisition module is configured to acquire an image sequence dataset of breast tissue.

[0049] A determination module is configured to input the image sequence dataset into a pre-established deep learning model to obtain a motion field between adjacent images; the deep learning model comprises a segmentation branch and a motion residual branch; the segmentation branch is configured to perform tissue segmentation on two consecutive images to obtain tissue segmentation data; the motion detection branch is configured to calculate motion change parameters of each pixel point in the two consecutive images; and the generation of the motion field is guided based on the tissue segmentation data and the motion change parameters.

[0050] A correction module is configured to correct the image sequence dataset based on the motion field to obtain a corrected image sequence dataset; and the corrected image sequence dataset is used to construct a three-dimensional image of the breast.

[0051] In a third aspect, the present application provides a computer device, which comprises a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the prone breast scanning motion artifact correction method of the first aspect or any of the corresponding embodiments thereof.

[0052] In a fourth aspect, the present application provides a computer readable storage medium storing computer instructions for causing a computer to execute the motion artifact correction method for prone scan according to the first aspect or any of the possible implementation forms thereof.

[0053] In a fifth aspect, the present application provides a computer program product comprising computer instructions for causing a computer to execute the motion artifact correction method for prone scan according to the first aspect or any of the possible implementation forms thereof.

[0054] It should be noted that the prone scan motion artifact correction device, the computer device, the computer readable storage medium and the computer program product provided by the present application correspond to the motion artifact correction method for prone scan described above. Therefore, the beneficial effects of the prone scan motion artifact correction device, the computer device, the computer readable storage medium and the computer program product are described above in the corresponding beneficial effects of the motion artifact correction method for prone scan, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the specific embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0056] Figure 1 is a flowchart of the motion artifact correction method for prone scan according to an embodiment of the present application;

[0057] Figure 2 is a schematic diagram of prone breast ultrasound scanning according to an embodiment of the present application;

[0058] Figure 3 is a schematic diagram of the corresponding motion vector of adjacent images according to an embodiment of the present application;

[0059] Figure 4 is a comparison schematic diagram of using motion field similarity before and after according to an embodiment of the present application;

[0060] Figure 5 is a schematic diagram of converting motion vector data into frequency domain motion data according to an embodiment of the present application;

[0061] Figure 6 is a comparison schematic diagram of motion field correction before and after according to an embodiment of the present application;

[0062] Figure 7 is a contrast schematic diagram of before and after artifact elimination according to an embodiment of the present application;

[0063] Figure 8 is a schematic diagram of breast three-dimensional reconstruction according to an embodiment of the present application;

[0064] Figure 9 is a schematic diagram of an image with noise according to an embodiment of the present application;

[0065] Figure 10 is a contrast schematic diagram of image similarity before and after noise filtering according to an embodiment of the present application;

[0066] Figure 11 is a schematic diagram of a segmentation process in a deep learning model according to an embodiment of the present application;

[0067] Figure 12 is a schematic diagram of a complete image after scanning according to an embodiment of the present application;

[0068] Figure 13 is a schematic diagram of an image after removing invalid areas according to an embodiment of the present application;

[0069] Figure 14 is a structural block diagram of a motion artifact correction device for prone breast scanning according to an embodiment of the present application;

[0070] Figure 15 is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0071] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings of the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0072] According to an embodiment of the present application, a motion artifact correction method for prone breast scanning is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0073] In this embodiment, a motion artifact correction method for prone breast scanning is provided, which can be executed by a server, a terminal, a mobile terminal, and the like, Figure 1is a flowchart of a motion artifact correction method for prone breast scanning according to an embodiment of the present application, as shown in Figure 1 The flow includes the following steps:

[0074] In step S101, a sequence of image data sets of breast tissue is obtained.

[0075] The present application is directed to a sequence of image data sets obtained after a prone breast scanning device scans breast tissue in a prone position. The image data sets can be ultrasound images or X-ray images, etc. As shown in Figure 2 The prone breast ultrasound scanning technology uses a cup-shaped structure, and the patient lies prone on the device, with the breast placed in the cup. The cup is filled with a selectable acoustic medium, such as water or other liquid or gas medium. An ultrasound probe rotates uniformly around the breast along the track of the cup, and two-dimensional ultrasound images of the breast tissue are collected, each containing angle information of the probe. The collected ultrasound images are encapsulated in sequence as data sets, thereby obtaining a sequence of image data sets.

[0076] In step S102, the sequence of image data sets is input into a pre-established deep learning model to obtain a motion field between adjacent images.

[0077] The motion field in this embodiment is used to describe the displacement of each pixel in the image over time. If the images in the sequence of image data sets are not affected by motion such as breathing, i.e., there is no motion, the difference between the motion fields of adjacent images is close to zero. Due to factors such as patient breathing, heart beating, muscle tension, etc., the position of the breast is displaced, resulting in a certain difference between the motion fields of adjacent two images. Assuming there are three images frame_n, frame_n+1, frame_n+2, in the presence of displacement, there is a certain difference between the motion fields of frame_n and frame_n+1 and between frame_n+1 and frame_n+2.

[0078] In this case, a deep learning model or other calculation method can be used to analyze the motion trend of adjacent images and calculate the motion vector between adjacent images. For example, for a 512x512 pixel image, the calculated motion vector can be a 512x512 motion vector data, which is used to express the motion relationship of the next image relative to the previous image. As shown in Figure 3 After the continuous 6 images are input into the model, the model outputs 5 motion vectors. The motion vector data includes two components, i.e., horizontal displacement (dx) and vertical displacement (dy).

[0079] The images in the image sequence dataset are continuous images. The continuous images are input into a pre-trained deep learning model. The model is trained to automatically learn and extract motion features in the images, especially the displacement and rotation information between objects or scenes. Finally, the model can calculate the motion field between each adjacent image, i.e., the motion vector of each pixel or region.

[0080] Regarding the establishment of the deep learning model:

[0081] Although the related art has proposed to estimate the motion field between ultrasound image frames by traditional image processing methods such as key point matching, in order to analyze the motion artifacts caused by patient breathing or probe movement, such methods are dependent on low-level features in the image, such as texture or corner matching. Such methods are greatly disturbed by factors such as image blur, noise, and tissue occlusion. Especially in breast ultrasound images, due to the softness and complex deformation of the tissue, the traditional key point matching method has obvious deficiencies in the continuity and accuracy of motion field estimation, and often cannot reflect the true tissue motion pattern.

[0082] To improve the correction accuracy of motion artifacts in breast ultrasound images and the generalization ability of the model, the present application makes improvements in the input, supervision mechanism and prediction target design of the deep learning model.

[0083] Regarding the input of the training data of the deep learning model, time domain information can be modeled by inputting continuous multiple frames of images. The so-called "continuous multiple frames of images" here are not limited to strictly adjacent frame sequences. In the specific embodiments of the present application, the image sequence can be: adjacent frames (such as frames t-1, t, t+1) continuously acquired; or frames sampled at a certain interval (such as frames t-2, t, t+2) to enhance the model's adaptability to different speed deformations; or mixed sampling (for example: t-2, t, t+1) to introduce asymmetric temporal context information.

[0084] Such interval sampling design can increase the diversity of training data and improve the robustness of the model to different motion speeds and deformation scales. Especially in ultrasound sequences of breast tissue, the breathing rates of different patients and the differences in breast structure are large, and the introduction of interval frames for training can effectively improve the generalization ability of the model. It can also enhance the model's ability to capture non-rigid deformation and improve the continuity and robustness of motion field estimation.

[0085] In addition, a structure supervision fusion mechanism is introduced in the deep learning model, which specifically includes:

[0086] The segmentation branch for auxiliary prediction is introduced to organize the partial image frames for segmentation prediction. The weak label information of the partial frames is used as a structural consistency supervision signal to improve the accuracy of the motion field in the edge region and the anatomical structure retention. The segmentation branch specifically processes the edge region of the image. Through accurate segmentation of the tissue edge data, it can help the deep learning network better understand the motion-induced deviation to effectively guide the model learning of the motion field with spatial structural consistency under the condition of large-scale unlabeled image data. The annotation data used only accounts for a very small proportion of the overall training data, thereby significantly reducing the cost of manual annotation. In addition, the annotation information can not only include the overall contour region of the breast, but also include some typical structure regions (such as glands, fat layers, ducts, etc.) inside the breast tissue to provide more detailed structural supervision.

[0087] A motion detection branch is also introduced to calculate the motion change parameters of each pixel point in consecutive two frames of images, including the motion direction and speed. In this way, the model can describe the motion of each pixel, especially the subtle changes in the position of the object in the image. These motion detections reflect the motion differences between adjacent frames, and the model uses these data to further refine the generation of the motion field. When generating the motion field, the model will use the edge information and the motion change parameters to guide the calculation of the motion field. That is, the above-mentioned edge information of the structure label is added to the multi-task loss function of the model as a supervision signal to prompt the model to learn the motion prediction result aligned with the anatomical structure until the training is completed, especially in the boundary region and the transition region of the tissue to help constrain the rationality of the motion field.

[0088] In actual use of the model, the image sequence data set is also input into the pre-established deep learning model, and then the edge of the image is segmented by the deep learning model to obtain the tissue segmentation data. At the same time, the motion change parameters of each pixel point in consecutive two frames of images are calculated, and the tissue segmentation data and the motion change parameters are used to guide the calculation of the motion field, so as to generate the final motion field and output. Thus, the accuracy of the motion field can be effectively improved, especially for scenes requiring high-precision motion estimation and edge protection, such as medical image analysis and video sequence processing.

[0089] In step S103, based on the motion field, the image sequence data set is corrected to obtain a corrected image sequence data set; the corrected image sequence data set is used to construct a three-dimensional breast image.

[0090] After the motion field is estimated by the calculation model, the image sequence dataset obtained by the acquisition can be corrected using the motion field, so that the image features of frame_n are closer to frame_n+1. By determining the motion field, the changes or displacements of each pixel point in the image between different frames can be estimated, and motion compensation or image alignment can be performed according to the changes or displacements, so that a corrected corrected image sequence dataset can be obtained.

[0091] After all the motion fields are obtained, all the motion fields are applied to the corresponding frames, and the similarity between the motion field and the adjacent frame is compared. It is found that after using the motion field, the proximity between the adjacent frame images is obviously improved. Referring to Figure 4 As shown, the red line part is the similarity comparison directly with the adjacent frame image, and the green line part is the similarity comparison after using the motion field with the adjacent frame image.

[0092] In this embodiment, the motion field based method for artifact correction can directly estimate the displacement vector between images, and according to the displacement vector, the image error caused by patient or device motion can be corrected more accurately. Compared with the traditional image registration method, the motion field can capture more local motion features and provide more detailed motion compensation. Moreover, the motion field correction can directly remove or reduce the motion artifacts from the image without relying on the alignment of the overall image. Especially in ultrasound images, the motion field correction can better handle dynamic changes such as periodic motion caused by breathing or heartbeat, thereby improving the accuracy of artifact correction.

[0093] In some optional embodiments, after obtaining the corrected image sequence dataset, the method further comprises:

[0094] Step S104, the motion vector data of all motion fields in the corrected image sequence dataset is counted.

[0095] Step S105, the motion vector data is converted to the frequency domain to obtain frequency domain motion data. Specifically, the data in the time domain can be converted to the frequency domain by applying Fourier transform or wavelet transform, etc., so as to obtain the frequency domain motion data.

[0096] Step S106, based on the frequency domain motion data, an abnormal motion pattern is determined.

[0097] Step S107, an abnormal motion field corresponding to the abnormal motion pattern is determined.

[0098] Step S108, the abnormal image corresponding to the abnormal motion field is corrected to obtain a target image sequence dataset.

[0099] Step S109, a breast three-dimensional image is constructed based on the target image sequence dataset.

[0100] A large number of ultrasound images can be pre-trained to extract common physiological motion patterns, such as periodic motion caused by breathing and heartbeat, and personalized pattern extraction can be performed for each patient. Specifically, the motion vector data of all motion fields can be counted, and then the counted motion vector data is converted to the frequency domain, referred to as shown in FIG. 6, and then the main motion patterns, such as periodic motion patterns such as breathing and heartbeat, are obtained through a model algorithm. Figure 5

[0101] Further, according to the extracted motion patterns, the images in the abnormal motion patterns deviating from the main motion patterns are corrected to eliminate motion artifacts and ensure the consistency and accuracy of the images. For example, the current scanning image of a breast has 1400 frames, and 1200 frames are extracted in the main motion patterns. Then, the remaining frames deviating from the main motion patterns, i.e., the corresponding influence in the abnormal motion patterns, are corrected to further obtain the target image sequence data.

[0102] In this embodiment, on the basis of correcting the image sequence data, the images deviating from the main motion patterns are corrected again through the recognition of the motion patterns, and the accuracy of the artifact correction is further improved.

[0103] In some optional embodiments, the abnormal motion pattern is determined based on the frequency domain motion data, including:

[0104] The frequency domain motion data is input into a pre-established motion pattern recognition model to determine the main motion pattern.

[0105] The abnormal motion pattern is determined based on the main motion pattern.

[0106] Specifically, a support vector machine, a neural network, or a clustering-based method can be used to establish a motion pattern recognition model. Through learning of a large amount of motion data, the model can recognize different motion patterns. That is, the frequency domain motion data is input into the model for feature extraction, such as extraction of energy, frequency peak value, amplitude, etc. in a specific frequency band, and according to the extracted features, the main motion patterns such as periodic motion caused by breathing and heartbeat are determined. After the model determines the main motion pattern, the abnormal motion pattern can be further determined. The abnormal motion pattern refers to other motion behaviors different from the main motion pattern, such as non-periodic motion caused by muscle tension and device vibration.

[0107] In this embodiment, the frequency domain data and the deep learning model are used to recognize the main motion pattern, which can effectively improve the accuracy of the main motion pattern recognition, thereby facilitating the correction accuracy of the abnormal images in the abnormal motion pattern.

[0108] In some optional embodiments, the abnormal images corresponding to the abnormal motion fields are corrected, including: ​

[0109] determining normal motion fields corresponding to normal motion patterns adjacent to the abnormal motion pattern;

[0110] correcting the abnormal image based on the normal motion fields and the abnormal motion field.

[0111] In order to correct the abnormal motion field, it is necessary to find motion fields adjacent to the abnormal motion field and conforming to the normal motion pattern. These normal motion fields refer to motion patterns showing consistency between adjacent frames and without significant deviation. Then, by comparing the motion fields between image frames, normal motion fields adjacent to the abnormal motion field in the time sequence are found. For example, assuming that the motion fields between frame2 and frame3 are normal and the motion fields between frame4 and frame5 are also normal, these normal motion fields can be selected as references for correction of the abnormal motion field, i.e. the motion field between frame3 and frame4 is corrected.

[0112] In this embodiment, by using this correction method, the motion fields of the image sequence can be ensured to be smoother and more consistent, thereby improving the accuracy of artifact correction.

[0113] In some optional embodiments, the abnormal image is corrected based on the normal motion fields and the abnormal motion field, including:

[0114] determining an average value of the normal motion fields and the abnormal motion field;

[0115] updating the normal motion fields based on the average value;

[0116] calculating displacement correction data based on the abnormal motion field and the average value;

[0117] correcting the abnormal image based on the displacement correction data.

[0118] Reference can be made to FIG. 1. Figure 6 As shown in FIG. 1, the abnormal motion pattern is determined, i.e. the motion field numbers deviating from the main motion pattern, such as frame3 (abnormal motion field) and the motion field x of frame4 (abnormal motion field) deviating. Then, a plurality of motion fields closest to the adjacent main motion pattern are obtained, such as the motion field y1 between frame2 and frame3 (normal motion field) and the motion field y2 between frame4 and frame5 (normal motion field). In order to enable smooth transition between frame2 and frame5, the images of frame3 and frame4 need to be re-corrected by the normal motion fields at this time.

[0119] Specifically, the average of the normal motion field and the abnormal motion field is first determined, i.e., the average of the motion field y1 between frame2 and frame3, the motion field x between frame3 and frame4, and the motion field y2 between frame4 and frame5. Frame3 is corrected by y1-mean(x, y1, y2), and frame4 is corrected by x-mean(x, y1, y2). Thus, the motion fields between frame2 and frame3, frame3 and frame4, and frame4 and frame5 are corrected to the average of x, y1, and y2. Referring to Figure 7 As shown, the motion-induced artifacts are effectively eliminated according to the method. Referring to Figure 8 As shown, the high-quality breast three-dimensional image generated by three-dimensional reconstruction of the image corrected by motion can effectively help doctors make more accurate diagnoses.

[0120] In this embodiment, by correcting the motion field in the image sequence, smooth transition between images can be ensured, and unnecessary motion deviation can be eliminated. The change of the motion field between adjacent frames conforms to the expected main motion pattern, avoiding unnatural jumps or mutations, thereby effectively improving the accuracy of three-dimensional reconstruction and providing more reliable image data support for doctors.

[0121] In an optional implementation, the deep learning model comprises a processing module for removing image noise, non-breast tissue regions, and regions opposite to the rotation center line.

[0122] Since the image sequence data set in this embodiment is acquired based on prone breast ultrasound scanning technology, and the derived medium is a low-viscosity liquid such as water. Using liquid medium as the sound-conducting medium will inevitably produce bubbles or water ripples caused by liquid internal motion during probe scanning, which will affect the quality of the collected images, as shown in Figure 9 As shown, the similarity between adjacent frames is relatively close, and if the motion field is directly determined by fitting adjacent frames, the reliable motion field will be difficult to determine due to the influence of noise. Referring to Figure 10 As shown, the image similarity (green line part) determined based on the image that has not been filtered by noise and corrected by the motion field cannot be higher than the similarity (red line part) between the images that have not been corrected.

[0123] To this end, the deep learning model used to determine the motion field in this embodiment is designed with a branch for shielding noise regions, so that the learning motion field only fits the breast region of adjacent frames. When learning the motion field using the deep learning model, the learned processing module is used to filter out the interfering parts first, and then the motion field vector is learned. The process of learning the motion field after filtering the noise by the processing module is shown in Figure 11 The parts using model inference in this embodiment mainly include two parts, i.e., breast region segmentation processing and motion field estimation processing. These two inference parts can be made by two deep learning models, or can be made in one model. In this embodiment, the mode of establishing two branches in one model is preferred for processing. The similarity between the images after noise filtering by the processing module and motion field correction is shown in Figure 4 It is shown that the shielding of the liquid medium image characteristics is increased, the more accurate motion field can be found, and the accuracy of the artifact correction is improved.

[0124] In this embodiment, the method not only improves the calculation efficiency, but also accurately captures the subtle motion changes between images, and can maintain high precision even in complex dynamic scenes. Through motion field correction, the quality of breast ultrasound images can be significantly improved, artifacts can be reduced, and the accuracy of three-dimensional reconstruction can be improved, providing more reliable image data support for doctors.

[0125] In this embodiment, the deep learning model used to determine the motion field is also designed with a unified image region screening mechanism to exclude all image regions that are considered "invalid" in the physical, imaging or structural aspects. Specifically, the water medium background region; no organizational structure information, which belongs to the noise region; non-breast tissue region, including the rib or lung structure appearing in the deep image; the region opposite to the rotation center line, which can be considered as redundant data of the current frame since the device will image from this angle again. Through joint image semantic recognition (such as image segmentation, anatomical structure recognition) and device imaging geometry knowledge (such as modeling of the rotation center axis).

[0126] This method performs region mask processing on the image before motion estimation, and only keeps the effective region for motion field calculation. Referring to Figure 12 , for the complete image after scanning, referring to Figure 13 , it is the image after removing the invalid region.

[0127] The region screening can be completed before the neural network or as a region mask input during network training to constrain the loss calculation region.

[0128] In an optional embodiment, the image sequence data set is input into a pre-established deep learning model to obtain the motion field between adjacent images, including:

[0129] obtaining a preliminary motion field between adjacent images determined by the deep learning model during the learning process;

[0130] obtaining a heartbeat time sequence corresponding to the acquisition of the image sequence dataset;

[0131] determining breast tissue periodic motion information based on the heartbeat time sequence;

[0132] performing consistency verification on the preliminary motion field based on the breast tissue periodic motion information;

[0133] in the case where the consistency verification is passed, outputting the corresponding motion field.

[0134] The deep learning model for determining the motion field in the embodiment is also designed with an auxiliary mechanism based on heartbeat signal modeling, aiming to improve the physiological reasonableness and physical consistency of the motion field estimated by the deep learning model.

[0135] Specifically, if the system has a synchronous electrocardiogram acquisition device, the heartbeat time sequence can be directly obtained, or the periodic analysis of the gray scale change in the key region (such as the chest wall near the chest wall) in the image sequence can be performed to indirectly obtain the heartbeat phase.

[0136] After obtaining the heartbeat time sequence, the periodic motion of the tissue driven by the heartbeat is modeled as a prior reference mode for the neural network learning as an additional input or a regularization term.

[0137] The consistency detection in the frequency domain or the phase domain is performed on the non-rigid motion field output by the neural network, and if an abnormal deformation obviously deviating from the heartbeat rhythm is found, re-estimation, post-processing correction or result filtering can be triggered.

[0138] The method helps to improve the consistency of the motion estimation result and the actual physiological motion, and the effect is significant especially when correcting the images of the deep part of the breast or the area close to the heart.

[0139] In an optional implementation, before obtaining the motion field between adjacent images, the method further includes:

[0140] obtaining relative motion information of the ultrasound probe between consecutive frames;

[0141] establishing a rigid motion geometric transformation model based on the relative motion information;

[0142] performing geometric transformation on each frame of image in the image sequence dataset based on the rigid motion geometric transformation model to obtain a motion-corrected image sequence dataset;

[0143] the motion-corrected image sequence dataset is taken as the image sequence dataset.

[0144] In this embodiment, the motion correction is performed before obtaining the motion field between adjacent images. The motion correction can be performed before inputting the deep learning model or after inputting the deep learning model, i.e., before the loss function learning after the model output. The motion correction is mainly used to eliminate the interference of known motion, so that the model can more easily obtain an accurate motion field.

[0145] In this embodiment, considering that the prone breast scanning device is a full-automatic three-dimensional scanning system, the ultrasound probe rotates or moves horizontally around the breast, and the relative pose is a known trajectory in the design stage or can be recorded in real time by a built-in sensor (such as an encoder or an IMU). Therefore, in this embodiment, the device motion influence can be removed from the image sequence through geometric modeling before the motion field estimation. Through the decoupling processing, the deep learning model for determining the motion field does not need to learn the rigid motion change caused by the device, and only needs to focus on the non-rigid motion deformation of the breast caused by the changes in breathing, heartbeat or body position, so that the motion field estimation is more accurate and stable, and the quality of subsequent artifact correction or three-dimensional reconstruction can be effectively improved.

[0146] In this embodiment, a motion artifact correction device for prone breast scanning is also provided. The device is used to implement the above-described embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible and is contemplated.

[0147] This embodiment provides a motion artifact correction device for prone breast scanning, as shown in Figure 14 The device includes:

[0148] The acquisition module 201 is configured to acquire an image sequence data set of breast tissue.

[0149] The determination module 202 is configured to input the image sequence data set into a pre-established deep learning model to obtain a motion field between adjacent images. The deep learning model includes a processing module configured to remove image noise, non-breast tissue regions, and regions on the opposite side of the rotation center line. The deep learning model includes a segmentation branch and a motion residual branch. The segmentation branch is configured to perform tissue segmentation on two consecutive images to obtain tissue segmentation data. The motion detection branch is configured to calculate the motion change parameters of each pixel point in the two consecutive images. The generation of the motion field is guided based on the tissue segmentation data and the motion change parameters.

[0150] The correction module 203 is configured to correct the image sequence dataset based on the motion field to obtain a corrected image sequence dataset, and the corrected image sequence dataset is used to construct a three-dimensional image of the breast. The correction module 203 is further configured to count motion vector data of all motion fields in the corrected image sequence dataset, convert the motion vector data to a frequency domain to obtain frequency domain motion data, determine an abnormal motion pattern based on the frequency domain motion data, determine a normal motion field corresponding to the abnormal motion pattern, correct an abnormal image corresponding to the abnormal motion field to obtain a target image sequence dataset, and construct a three-dimensional image of the breast based on the target image sequence dataset. The correction module 203 is further configured to determine a normal motion field corresponding to a normal motion pattern adjacent to the abnormal motion pattern, correct the abnormal image based on the normal motion field and the abnormal motion field, determine an average value of the normal motion field and the abnormal motion field, calculate displacement correction data based on the abnormal motion field and the average value, and correct the abnormal image based on the displacement correction data.

[0151] The correction module 203 is further configured to obtain relative motion information of the ultrasound probe between consecutive frames, establish a rigid motion geometric transformation model based on the relative motion information, perform geometric transformation on each frame of image in the image sequence dataset based on the rigid motion geometric transformation model to obtain a motion-corrected image sequence dataset, and use the motion-corrected image sequence dataset as the image sequence dataset.

[0152] The correction module 203 is further configured to obtain a preliminary motion field between adjacent images determined by the deep learning model in a learning process, obtain a corresponding heartbeat time sequence when the image sequence dataset is collected, determine breast tissue periodic motion information based on the heartbeat time sequence, perform consistency verification on the preliminary motion field based on the breast tissue periodic motion information, and output a corresponding motion field in a case where the consistency verification is passed.

[0153] The motion artifact correction device for prone breast scanning in the embodiment is presented in the form of functional units, and the units refer to ASIC circuits, processors and memories executing one or more software or fixed programs, and / or other devices that can provide the above functions.

[0154] Further function descriptions of the above modules and units are the same as those of the above corresponding embodiments, and will not be described here.

[0155] The embodiment of the present application further provides a computer device having the above Figure 14 The motion artifact correction device for prone breast scanning shown in the figure.

[0156] Please refer to Figure 15 , Figure 15 is a structural schematic diagram of a computer device provided by an optional embodiment of the present application, as shown in Figure 15As shown, the computer device includes one or more processors 10, memory 20, and interfaces 30 for external devices such as a keyboard and a mouse and peripheral devices such as disk devices or other storage devices. One or more busses 10 can be used to implement the interface between the various internal and external components and can be implemented using any one or more of a variety of bus technologies including a System bus, PCI, SCSI, AGP, Super- I / O bus, etc. Furthermore, various buses can be used in front side buses, back side buses, and other bus configurations based on any bus or messaging technology known to those skilled in the art. Figure 15 The processor 10 is used in the embodiments below as an example.

[0157] The processor 10 can be a central processing unit, a network processor, or a combination thereof. The processor 10 can further include a hardware chip. The hardware chip can be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device can be a complex programmable logic device, a field programmable logic device, a general array logic, or any combination thereof.

[0158] The memory 20 stores instructions that can be executed by the at least one processor 10, so that the at least one processor 10 implements the method shown in the above embodiments.

[0159] The memory 20 can include a program region and a data region. The program region can store an operating system and an application program required by at least one function. The data region can store data created according to the use of the computer device, etc. In addition, the memory 20 can include a high-speed random access memory, and can further include a non-transitory memory such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some alternative embodiments, the memory 20 can optionally include a memory disposed remotely with respect to the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0160] The memory 20 can include a volatile memory such as a random access memory, and can also include a non-volatile memory such as a flash memory, a hard disk, or a solid state disk. The memory 20 can further include a combination of the above-mentioned kinds of memories.

[0161] The computer device further includes a communication interface 30 for communication with other devices or communication networks.

[0162] The embodiments of the present application further provide a computer readable storage medium, and the method according to the embodiments of the present application can be implemented in hardware, firmware, or recorded in a storage medium, or stored in a remote storage medium or a non-transitory machine readable storage medium and downloaded to a local storage medium through network, so that the method described herein can be processed by such software on a storage medium using a general purpose computer, a special purpose processor, or programmable or special hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid state disk, etc. Further, the storage medium can also include a combination of the above-mentioned memories. It can be understood that the computer, the processor, the microprocessor controller, or the programmable hardware includes a storage component that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the method shown in the above embodiments is implemented.

[0163] Part of the present application can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, the method and / or technical solutions according to the present application can be called or provided. Those skilled in the art should understand that the form of computer program instructions in computer readable medium includes but is not limited to source file, executable file, installation package file, etc. Correspondingly, the way of computer program instructions executed by computer includes but is not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.

[0164] Although the embodiments of the present application are described in conjunction with the accompanying drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A method of motion artifact correction for prone breast scans, the method comprising: The method comprises: acquiring an image sequence data set of breast tissue; inputting the image sequence data set into a pre-established deep learning model to obtain a motion field between adjacent images; the deep learning model comprises a segmentation branch and a motion residual branch; the segmentation branch is used for region segmentation of two consecutive images to obtain region segmentation data; the motion detection branch is used for calculating motion change parameters of each pixel point in two consecutive images; and generation of the motion field is guided based on the tissue segmentation data and the motion change parameters; based on the motion field, correcting the image sequence data set to obtain a corrected image sequence data set; the corrected image sequence data set is used for constructing a three-dimensional image of the breast.

2. The method of claim 1, wherein, After the corrected image sequence data set is obtained, the method further comprises: counting motion vector data of all the motion fields in the corrected image sequence data set; converting the motion vector data to a frequency domain to obtain frequency domain motion data; based on the frequency domain motion data, determining an abnormal motion pattern; determining an abnormal motion field corresponding to the abnormal motion pattern; correcting an abnormal image corresponding to the abnormal motion field to obtain a target image sequence data set; constructing the three-dimensional image of the breast based on the target image sequence data set.

3. The method of claim 2, wherein, The method for determining an abnormal motion pattern based on the frequency domain motion data comprises: inputting the frequency domain motion data into a pre-established motion pattern recognition model to determine a main motion pattern; based on the main motion pattern, determining the abnormal motion pattern.

4. The method of claim 2, wherein, The method for correcting the abnormal image corresponding to the abnormal motion field comprises: determining a normal motion field corresponding to a normal motion pattern adjacent to the abnormal motion pattern; based on the normal motion field and the abnormal motion field, correcting the abnormal image.

5. The method of claim 4, wherein, The method for correcting the abnormal image based on the normal motion field and the abnormal motion field comprises: determining an average value of the normal motion field and the abnormal motion field; based on the abnormal motion field and the average value, calculating displacement correction data; based on the displacement correction data, correcting the abnormal image.

6. The method of claim 1, wherein, The deep learning model comprises a processing module for removing image noise, non-breast tissue regions, and rotation center line opposite regions.

7. The method of claim 1, wherein, The method for inputting the image sequence data set into a pre-established deep learning model to obtain a motion field between adjacent images comprises: acquiring a preliminary motion field between adjacent images determined by the deep learning model during a learning process; acquiring a corresponding heartbeat time sequence when the image sequence data set is collected; based on the heartbeat time sequence, determining breast tissue periodic motion information; based on the breast tissue periodic motion information, performing consistency verification on the preliminary motion field; in the case of passing the consistency verification, outputting a corresponding motion field.

8. The method of claim 1, wherein, Before the motion field between adjacent images is obtained, the method further comprises: acquiring relative motion information of an ultrasonic probe between consecutive frames; based on the relative motion information, establishing a rigid motion geometric transformation model; Based on the rigid motion geometric transformation model, each frame of image in the image sequence dataset is geometrically transformed to obtain a motion-corrected image sequence dataset; The motion-corrected image sequence dataset is used as the image sequence dataset.

9. An apparatus for motion artifact correction of prone breast scans, characterized in that, The device comprises: An acquisition module is configured to acquire an image sequence dataset of breast tissue; A determination module is configured to input the image sequence dataset into a pre-established deep learning model to obtain a motion field between adjacent images; the deep learning model comprises a segmentation branch and a motion residual branch; the segmentation branch is configured to perform tissue segmentation on two consecutive images to obtain tissue segmentation data; the motion detection branch is configured to calculate motion change parameters of each pixel point in the two consecutive images; and the generation of the motion field is guided based on the tissue segmentation data and the motion change parameters; A correction module is configured to correct the image sequence dataset based on the motion field to obtain a corrected image sequence dataset; and the corrected image sequence dataset is used to construct a three-dimensional image of the breast.

10. A computer device, comprising: It comprises: A memory and a processor are communicatively connected to each other, and the memory stores computer instructions; the processor executes the computer instructions to perform the motion artifact correction method for prone breast scanning according to any one of claims 1-8.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the computer to execute the motion artifact correction method for prone breast scanning according to any one of claims 1-8.

12. A computer program product, characterised in that, The computer instructions are used to cause the computer to execute the motion artifact correction method for prone breast scanning according to any one of claims 1-8.

Images