Breast ultrasonic image splicing method, device, equipment, medium and product

Abstract

The invention discloses a mammary gland ultrasound image splicing method and device, equipment, a medium and a product, and relates to the field of image processing, and the method comprises the steps: obtaining a resampling image based on a coronal plane image; nipple positioning is carried out according to the resampling image, and a nipple contour and nipple center coordinates are obtained; mapping the nipple contour and the nipple center coordinate to the coronal plane image to obtain a nipple center line; calculating deviation angles of different body positions relative to the median position according to the direction vector of the center line of the nipple; determining a splicing reference point of all coronal plane images in each body position according to an intersection point of the center line of the nipple and the coronal plane section frame number of the coronal plane images; calculating and removing overlapping areas of coronal plane images and median images of different body positions according to the splicing datum points; and performing longitudinal offset compensation according to the offset angle and the coronal plane image of which the overlapping region is removed, and then performing splicing to obtain a panoramic image of the mammary gland. According to the method, rapid, non-repeated and smooth splicing of the breast ultrasound images can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to a breast ultrasound image splicing method, device, equipment, medium and product. BACKGROUND

[0002] At present, breast imaging examinations mainly include mammography, breast ultrasound and breast MRI. However, due to the fact that the breasts of Asian women are generally dense and small in size, the sensitivity and specificity of mammography for breast cancer screening in Asian women are both low. At the same time, MRI examination is expensive and time-consuming. According to the China Female Breast Cancer Screening Guidelines (2022 Edition), for Chinese women, breast MRI examination is currently only suitable for supplementary examination of high-risk women. In contrast, ultrasound imaging examination can effectively distinguish between lumps and normal tissues in dense breast tissue, and has the advantages of no radiation risk, high safety, real-time generation of dynamic images, low price, etc., and therefore becomes the most important medical imaging method for large-scale breast cancer screening in China. However, ordinary ultrasound examination also has some limitations. For example, the scanning field of view of handheld ultrasound is limited, and only a part of the breast can be scanned each time, which may result in the omission of some lesions. At the same time, the quality of the scanning results also depends on the technical level of the operator, and the experience and skills of different operators can significantly affect the quality and consistency of the examination results.

[0003] In order to overcome the limitations of traditional handheld ultrasound, ABUS (Automated Breast Ultrasound System) imaging technology has emerged. This system uses an arc-shaped ultra-wideband probe, has a wider field of view and covers more areas. At the same time, it adopts new working modes such as standard full-volume, automatic scanning and three-dimensional reconstruction, realizes automatic and rapid image acquisition, and can be standardized. This greatly overcomes the dependence of handheld ultrasound on operator experience, ensures uniform coverage of all areas of the breast, reduces the risk of lesion omission, and has high detection rate and accuracy. However, due to the characteristics of multi-position scanning, the ABUS system generates a large number of images from different positions, and the doctor needs to check each image to find suspicious lesion areas, and often faces a large amount of redundant image information when analyzing. For example, a breast area may appear in multiple position images, and the doctor needs to determine which information is redundant and which information is valuable. Such redundant information not only increases the burden of the doctor, but also easily causes visual fatigue, and may also cause delay or error in the diagnosis result. Therefore, multi-position splicing technology has great significance in ABUS breast cancer screening.

[0004] Multi-position splicing technology can intelligently fuse multiple images from different positions to form a continuous and full-view breast panorama image. It eliminates redundant information and concentrates key diagnostic information. Meanwhile, the intuitive panorama image enables doctors to quickly lock suspicious areas, effectively shortening the doctor's reading time and improving the diagnostic efficiency, thereby reducing the burden of manual screening. This greatly optimizes the breast ultrasound examination process and provides strong technical support for early detection and accurate diagnosis of breast cancer.

[0005] In ABUS breast ultrasound examination, since different positions and angles affect the anatomical structure in the image, it is particularly important to have a quick, non-repetitive and smooth image splicing. SUMMARY

[0006] The purpose of the present application is to provide a breast ultrasound image splicing method, device, equipment, medium and product, which can realize quick, non-repetitive and smooth splicing of breast ultrasound images.

[0007] To achieve the above purpose, the present application provides the following solutions: In a first aspect, the present application provides a breast ultrasound image splicing method, comprising: selecting coronal plane images in breast ultrasound data of different positions of the same breast and obtaining a resampling image based on the coronal plane images; performing nipple positioning according to the resampling image to obtain a nipple contour and a nipple center coordinate; mapping the nipple contour and the nipple center coordinate to the coronal plane image to obtain a nipple center line; calculating an offset angle of different positions relative to the median position according to a direction vector of the nipple center line; determining a splicing reference point of all coronal plane images in each position according to the intersection of the nipple center line and the coronal plane image coronal plane section frame number; calculating an overlapping area of coronal plane images of different positions and the median position image according to the splicing reference point and removing the overlapping area in the coronal plane image; performing longitudinal offset compensation according to the offset angle and the coronal plane image with the removed overlapping area; splicing the coronal plane image after longitudinal offset compensation to obtain a panorama image of the breast.

[0008] In a second aspect, the present application provides a breast ultrasound image splicing device, comprising: a selection module configured to select coronal plane images in breast ultrasound data of different positions of the same breast and obtain a resampling image based on the coronal plane images; a nipple positioning module configured to perform nipple positioning based on the resampled image to obtain a nipple contour and a nipple center coordinate; a mapping module configured to map the nipple contour and the nipple center coordinate to the coronal plane image to obtain a nipple center line; an offset angle calculation module configured to calculate an offset angle of different body positions relative to a median position based on a direction vector of the nipple center line; a stitching reference point calculation module configured to determine a stitching reference point of all coronal plane images in each body position based on an intersection of the nipple center line and a frame number of a coronal plane section of the coronal plane image; an overlapping region removal module configured to calculate an overlapping region of the coronal plane image of different body positions and the median position image based on the stitching reference point and remove the overlapping region in the coronal plane image; a longitudinal offset compensation module configured to perform longitudinal offset compensation based on the offset angle and the coronal plane image with the removed overlapping region; a stitching module configured to stitch the coronal plane image after the longitudinal offset compensation to obtain a panoramic image of the breast.

[0009] In a third aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the breast ultrasound image stitching method.

[0010] In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the breast ultrasound image stitching method.

[0011] In a fifth aspect, the present application provides a computer program product comprising a computer program, wherein the computer program is executable by a processor to implement the breast ultrasound image stitching method.

[0012] According to the embodiments provided in the present application, the following technical effects are disclosed: The breast ultrasound image splicing method, device, equipment, medium and product provided by the application map the nipple contour and the nipple center coordinates to the coronal plane image to obtain a nipple center line; calculate an offset angle of different body positions relative to the median position according to a direction vector of the nipple center line; determine a splicing reference point of all coronal plane images in each body position according to the intersection of the nipple center line and the coronal plane image coronal plane section frame number, and through fitting of the overall nipple center, the splicing reference point of each frame of different body position images can be quickly and accurately determined at one time; calculate an overlapping area of the coronal plane images of different body positions and the median position image according to the splicing reference point, and remove the overlapping area in the coronal plane image; perform longitudinal offset compensation according to the offset angle and the coronal plane image with the removed overlapping area; splice the coronal plane image after longitudinal offset compensation to obtain a panoramic image of the breast; the overlapping area is determined and removed through the splicing reference point, and the non-overlapping splicing of the image can be realized; the longitudinal offset compensation can ensure the smoothness of the spliced image. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0014] Figure 1 The flowchart of the breast ultrasound image splicing method in actual application.

[0015] Figure 2 The flowchart of two-dimensional breast ultrasound nipple positioning.

[0016] Figure 3 The schematic diagram of edge detection.

[0017] Figure 4 The schematic diagram of removing large-area and small-area closed figures.

[0018] Figure 5 The schematic diagram of recording the coordinate information of the maximum figure of roundness.

[0019] Figure 6 The network structure diagram of the deep learning segmentation algorithm U2Net.

[0020] Figure 7 The schematic diagram of positioning the nipple by the deep learning segmentation algorithm.

[0021] Figure 8 Mapping the nipple contour and the center coordinates to the original coronal plane ultrasound image.

[0022] Figure 9 This is a multi-position stitched image.

[0023] Figure 10 The flowcharts show the traditional nipple detection algorithm and the multi-position stitching method based on gradient weighted fusion.

[0024] Figure 11 This is a flowchart illustrating a breast ultrasound image stitching method provided in another embodiment of this application.

[0025] Figure 12 This is a schematic diagram of the functional modules of a breast ultrasound image stitching device provided in an embodiment of this application.

[0026] Figure 13 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] In ABUS breast ultrasound examinations, different body positions and angles affect the anatomical structures in the images. Therefore, finding a suitable stitching feature matching point is particularly important for multi-position stitching. The nipple is a highly prominent feature in breast anatomy; it is usually the highest point of the breast and has a nearly round shape. Compared to other breast tissue structures (such as glandular or adipose tissue), the nipple presents a more distinct boundary in ultrasound images, making it easier for algorithms to detect. Therefore, using the nipple as a central and fixed anatomical landmark can provide a common reference point for images from different positions, ensuring alignment and matching between images, thereby achieving accurate and continuous multi-position stitching.

[0029] Existing techniques for ABUS multi-position stitching first use image processing algorithms to identify and label nipple locations in ABUS images from different positions. Secondly, the identified and labeled nipple locations are designated as feature points for subsequent stitching algorithms. Then, the images to be stitched are fitted and transformed by matching these feature points. Finally, gradient weighting is used to overlay and fuse the images. However, image processing-based nipple recognition algorithms rely on threshold selection; fixed processing steps cannot adapt well to various ultrasound images. While gradient weighting stitching can effectively reduce discontinuities in the stitched panoramic image, the presence of noise and artifacts in ultrasound images can lead to uneven brightness or abrupt texture changes in boundary areas. Furthermore, repeatedly locating nipples in the coronal plane at all depths is computationally complex and time-consuming.

[0030] A traditional nipple detection algorithm includes the following steps: first, the image is binarized; then, opening / closing and inversion operations are performed on the binarized image to detect circular regions resembling nipples; finally, white areas connected to the image border and white targets within a threshold area (i.e., small white targets) are deleted, leaving the final detected target, the nipple. This method uses only some basic image processing operations, such as noise reduction, binarization, opening / closing, and inversion operations. These techniques are computationally simple and require less resources and time. However, breast images from different patients vary greatly, and a single threshold and fixed steps cannot adapt to all situations. This fixed process lacks adaptability, leading to unstable results in practical applications, especially when dealing with highly variable ultrasound images.

[0031] Another multi-position stitching method based on gradient-weighted fusion includes the following steps: First, the mask overlap regions of each coronal ultrasound image in the image to be fused are extracted. Then, a weighted mask for the corresponding coronal ultrasound image is constructed based on the mask overlap region. Following a gradient-weighted approach, each coronal ultrasound image is mapped to the final stitched panoramic image. Finally, the overlap regions of the mapped coronal ultrasound images are fused together. The weighted mask can dynamically adjust the weights according to the characteristics and requirements of different images, allowing high-quality images to occupy a larger proportion in the fusion process. This effectively reduces boundary discontinuities between different images, thereby improving the clarity of the final image. However, when faced with ultrasound images containing significant noise and artifacts, the gradient-weighted fusion method may mistakenly identify noisy parts as meaningful details, directly affecting the accuracy of the overlap region and the weight allocation, resulting in unsatisfactory fusion results. Furthermore, for ABUS images with hundreds of frames, the gradient-weighted fusion method can only stitch images from different positions onto a single frame at a time. Processing all frames of ultrasound images requires lengthy computation times, hindering efficient and rapid image review by physicians.

[0032] This application combines image processing algorithms with deep learning models to accurately locate the nipple region and determine its center coordinates on coronal section images. Based on the nipple localization results on the superficial coronal plane, this technology can fit the nipple centerline, and then calculate the coordinates of the stitching reference points for all coronal planes under different body positions, as well as the offset angle of each body position relative to the midline. Based on these stitching reference point coordinates, this technology can calculate the overlapping areas of all coronal plane images in different body positions in one complete operation, and remove the overlapping areas and compensate for the offset angles. Finally, the non-overlapping and compensated images from multiple body positions are integrated and stitched into a complete, smooth, and continuous panoramic image of the breast. This invention enables doctors to assess the overall health of the breast more quickly and intuitively, significantly reducing the workload of doctors and significantly improving the efficiency and accuracy of diagnosis.

[0033] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] In one exemplary embodiment, such as Figure 11 As shown, a method for stitching breast ultrasound images is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is described using a server as an example, and includes the following steps.

[0035] Step 1101: Select coronal images from breast ultrasound data of different positions of the same breast and obtain resampled images based on the coronal images.

[0036] Step 1102: Perform nipple localization based on the resampled image to obtain the nipple contour and nipple center coordinates.

[0037] Step 1103: Map the nipple contour and the nipple center coordinates onto the coronal image to obtain the nipple centerline.

[0038] Step 1104: Calculate the offset angle of different body positions relative to the midline based on the direction vector of the nipple centerline.

[0039] Step 1105: Determine the stitching reference point for all coronal images in each position based on the intersection of the nipple center line and the number of frames of the coronal section of the coronal image.

[0040] Step 1106: Calculate the overlapping area between the coronal images and the median images in different body positions based on the stitching reference points, and remove the overlapping area in the coronal images.

[0041] Step 1107: Perform longitudinal offset compensation based on the offset angle and the coronal image with the overlapping area removed.

[0042] Step 1108: Stitch together the coronal images after longitudinal offset compensation to obtain a panoramic image of the breast.

[0043] By fitting the overall nipple center, the fitted nipple centerline can be used to quickly and accurately determine the stitching reference point for each frame of images in different positions. Based on the stitching reference point, the overlapping area between the coronal and median images in different positions is calculated, and the overlapping area in the coronal image is removed. Vertical offset compensation is performed based on the offset angle and the coronal image with the overlapping area removed. The longitudinally offset-compensated coronal images are then stitched together to obtain a panoramic image of the breast. By determining and removing the overlapping area using the stitching reference point, non-overlapping image stitching can be achieved. Vertical offset compensation ensures the smoothness of the stitched image.

[0044] In an exemplary embodiment, selecting coronal images from breast ultrasound data of different positions of the same breast and obtaining resampled images based on the coronal images specifically includes: sequentially selecting coronal images of different depths from breast ultrasound data of different positions of the same breast along the coronal plane; and resampling the coronal images using bilinear interpolation to obtain resampled images.

[0045] In an exemplary embodiment, nipple localization is performed based on the resampled image to obtain the nipple contour and nipple center coordinates. Specifically, this includes: determining the region containing the nipple based on the resampled image using an image processing algorithm or a deep learning segmentation algorithm; performing edge detection based on the region containing the nipple using the Sobel edge detection operator to obtain the nipple contour; and calculating the centroid based on the nipple contour to obtain the nipple center coordinates.

[0046] In an exemplary embodiment, mapping the nipple contour and the nipple center coordinates to the coronal image to obtain the nipple centerline specifically includes: mapping the nipple contour and the nipple center coordinates to the coronal image to obtain a coordinate sequence of the nipple center; and fitting the coordinate sequence of the nipple center using a curve fitting method to obtain the nipple centerline.

[0047] This application combines image processing algorithms with deep learning models to accurately locate the nipple and calculate its center coordinates on coronal section images. Simultaneously, based on the obtained superficial nipple center coordinates, this technology can fit the nipple centerline and offset angle under different body positions, thereby determining the stitching reference point for each frame of the coronal section image in different positions. Overlapping areas are calculated and removed based on the distance from the stitching reference point to the image boundary, and corresponding offset compensation is performed according to the offset angle. This achieves rapid, non-overlapping, smooth, and complete panoramic breast image stitching from multi-position scanning using automated breast ultrasound. This technology effectively reduces the workload of physicians processing images from multiple positions during automated breast volumetric ultrasound image interpretation, greatly improving the efficiency of breast cancer screening and diagnosis.

[0048] This application aims to provide an automated breast volume ultrasound multi-position stitching technology based on nipple positioning, used to automatically stitch together smooth ABUS images without overlapping areas. This addresses the problem of low efficiency and decreased accuracy in image interpretation caused by doctors having to repeatedly read images of the same breast from different positions; it also overcomes the limitation of ABUS breast ultrasound images in terms of the limited field of view, which prevents a complete view of the entire breast region. To achieve the above objectives, this application adopts the following technical solution, such as... Figure 1 and Figure 10 As shown: 1) Scan the same breast from three positions: lateral (LLAT or RLAT), midline (LAP or RAP), and medial (LMED or RMED) to obtain ABUS scan data for the three positions.

[0049] 2) From the ABUS scan data of three positions, coronal images at different depths were selected sequentially along the coronal plane for subsequent calculations.

[0050] In practical applications, ABUS breast ultrasound data of a single patient, with a size of 500×400×1596, is sliced ​​along the coronal plane to obtain coronal breast ultrasound images of the left breast from three positions, totaling 1200 images, each with a size of 500×1596. The three positions are LMED (left medial), LAP (left midline), and LLAT (left lateral), and their images are denoted as follows: .

[0051] 3) Resample the selected coronal images using bilinear interpolation. × The size of the resampled image is obtained. ; , where N is the number of pixels in the horizontal width of the selected coronal ultrasound image.

[0052] In practical applications, each 500×1596 coronal section breast ultrasound image is resampled to 500×500 using bilinear interpolation to obtain the resampled image. .

[0053] 4) In resampled images Nipple localization is performed, such as... Figure 2 As shown, there are two methods available for this step: Option 1: Image processing algorithm.

[0054] a. Selecting a threshold For resampled images Perform binarization processing to obtain a binarized image. ;in The selection range is: .

[0055] In practical applications, =0.5.

[0056] b. For binarized images Invert the value of each pixel to obtain the inverted image. .

[0057] .

[0058] c. For example Figure 3 As shown, the Canny edge detection algorithm is used to process the inverted image. Extract edge information to obtain an image containing edge information. Among them, the Canny edge detection algorithm is a commonly used image edge detection algorithm in this field, such as... Figure 3 As shown.

[0059] d. For images containing edge information The closed graphic region in the image is filled to obtain the filled image. .

[0060] e. Remove the filled image medium area smaller Closed figures and areas greater than The closed figure is used to obtain the image. ,like Figure 4 As shown. Among them. and These are the set small-area threshold and large-area threshold, respectively. In practical applications, , .

[0061] f. Calculate the image Roundness of each closed shape : .

[0062] in For image The Middle A closed figure, For the first The area of ​​a closed figure. For the first The perimeter of a closed figure. In practical applications, the roundness of each closed figure. ={0.48, 0.23, 0.16}.

[0063] g. Record roundness The largest closed shape is the nipple region, resulting in a shape containing the nipple region. like Figure 5 As shown.

[0064] Option 2: Deep learning segmentation algorithm: a. Constructing the deep learning segmentation network U2Net. In practical applications, the deep learning segmentation network U2Net is constructed based on frameworks such as PyTorch and NumPy. The network structure is as follows: Figure 6 As shown.

[0065] b. Using coronal section images of 6 positions from 80 patients collected independently, a nipple localization dataset containing 1920 images was formed.

[0066] c. Divide the nipple localization dataset into a training set (1440 images) and a test set (480 images) in an 8:2 ratio.

[0067] d. During training, the optimal deep learning segmentation network weights are obtained by optimizing the model hyperparameters and training strategy. In practical applications, the batch size is set to 8, Adam is selected as the network optimizer, the learning rate is set to 0.007, and the optimal deep learning segmentation network weights are obtained through 200 training rounds.

[0068] e. Based on the determined optimal weights, evaluate the network model on the test set. If its average Dice coefficient ≥ 0.9 and average IoU value ≥ 0.8, then the model is considered a satisfactory nipple localization network. In practical applications, based on the obtained optimal weights, the network model is evaluated on the test set. Its average Dice is 0.9023 and its average IoU is 0.875, therefore the model is considered a satisfactory nipple localization network.

[0069] f. Transfer the image The input is fed into a trained nipple localization network to obtain images containing nipple regions. like Figure 7 As shown.

[0070] 5) Take the result from step 4) Provided to the host computer interface, for manual judgment. Does it contain correct nipple location information?

[0071] 6) If not included, the image will be resampled. The information is provided to the host computer interface, where the nipple location information is manually marked and a correct value is regenerated. If included, please specify. Correctly identify the nipple area.

[0072] 7) Use the Sobel edge detection operator to pair Edge detection is performed to extract the nipple contour. The centroid (the average value of the edge point coordinates) is calculated using the extracted set of edge point coordinates to determine the nipple center coordinates. ,in Indicates the number of frames. and These represent the two-dimensional coordinates of the nipple center at a certain position within the frame.

[0073] 8) The obtained nipple contour and center point are mapped onto the original coronal ultrasound image using bilinear interpolation, such as... Figure 8 As shown.

[0074] 9) By repeating the above steps, the coordinate sequence of the nipple center in multiple coronal section images can be obtained, as shown below: .

[0075] .

[0076] in, They represent the first Nipple center coordinates in medial, medial, and lateral coronal sections. ,in This refers to the maximum number of frames in the coronal section ultrasound images taken from a specific patient position. The effective number of frames actually involved in constructing the nipple center coordinate sequence. Let x be the two-dimensional x-axis coordinate of the nipple center in the i-th frame of the lateral view. Let y be the two-dimensional coordinate of the nipple center in the i-th frame of the lateral view. Let x be the two-dimensional x-axis coordinate of the nipple center in the i-th frame of the midline image. Let y be the two-dimensional coordinate of the nipple center in the i-th frame of the midline image. Let x be the two-dimensional x-axis coordinate of the nipple center in the i-th frame of the medial view. Let y be the two-dimensional coordinate of the nipple center in the i-th frame of the medial view. Let i be the frame number (depth parameter) corresponding to the image. Among them, the coronal section image is the coronal image in (2).

[0077] In practical applications, frames 50, 60, and 70 on the left side are selected as nipple centerline fitting frames, and the nipple coordinates in different body positions in these three frames are obtained through the above image processing algorithm or deep learning algorithm.

[0078] The nipple center coordinates of the LLAT, LAP, and LMED positions in frame 50 are (282, 241), (811, 243), and (1326, 245), respectively.

[0079] The nipple center coordinates of the LLAT, LAP, and LMED positions in frame 60 are (285, 242), (812, 242), and (1324, 241), respectively.

[0080] The nipple center coordinates of the LLAT, LAP, and LMED positions in frame 70 are (289, 241), (811, 241), and (1320, 243), respectively.

[0081] 10) For the coordinate sequence of the nipple center in each body position, fit the nipple centerline for each body position using a curve fitting method (default least squares method). The objective function and the fitted equation for the nipple centerline are expressed as follows: .

[0082] .

[0083] .

[0084] .

[0085] in, These represent the equations of the nipple centerlines for the lateral, midline, and medial positions, respectively, with their direction vectors being ( ), ( ), ( ); These are fixed points on the center line of the nipple. x0 is the initial intercept of the fitted line on the x-axis (x-coordinate of the line's starting point); y0 is the initial intercept of the fitted line on the y-axis (y-coordinate of the line's starting point). To fit the slope of the straight line in the x-direction, To fit the slope of the straight line in the y-direction; is the frame number (depth parameter) corresponding to the i-th frame image; n is the total number of frames involved in the fitting (n=3, corresponding to frames 50, 60, and 70). T is a parameter variable (which can be understood as the "step size" or "distance parameter" along the straight line), used to describe the position of any point on the straight line; The x-coordinate of the starting point of the center line of the lateral (LAT) nipple; (The y-coordinate of the starting point of the lateral (LAT) nipple centerline). The number of frames (depth) corresponding to the starting point of the lateral (LAT) nipple centerline. The direction vector component of the lateral (LAT) nipple centerline in the x-direction; The y-direction vector component of the lateral (LAT) nipple centerline; The direction vector component (set to 1) of the lateral (LAT) nipple centerline in the z-direction (frame number / depth).

[0086] In practical applications, based on the nipple coordinates obtained in frames 50, 60, and 70, the least squares method is used to fit the nipple centerline equations for different body positions. : .

[0087] 。

[0088] .

[0089] Their direction vectors are respectively .

[0090] 11) Based on the nipple center line The direction vector is used to calculate the offset angle relative to the midline position for different body positions (lateral position, medial position). In an exemplary embodiment, the offset angle includes a first offset angle. Second offset angle The formula for calculating the offset angle is as follows: .

[0091] .

[0092] in, The x-component of the direction vector of the lateral nipple centerline. The x-component of the direction vector of the median nipple centerline is given. The y-component of the direction vector of the lateral nipple centerline is given. The y-component of the direction vector of the median nipple centerline is given. The z-component of the direction vector of the lateral nipple centerline is given. The z-component of the direction vector of the median nipple centerline is given. Let x be the x-component of the direction vector of the medial nipple centerline. The y-component of the direction vector of the medial nipple centerline is given. The z-component of the direction vector of the medial nipple centerline.

[0093] In practical applications, , .

[0094] 12) Number of frames through the nipple centerline and coronal section The intersection points are used to determine the stitching reference points for all coronal section ultrasound images in each position (lateral, midline, medial). The calculation formula is as follows: .

[0095] in, Indicates the number of frames. . For body position index, Lateral position, It is the medial position. It is the center position. Let y be the y-coordinate of the stitching reference point on the i-th frame of the coronal image under body position k. Let y be the coronal coordinate of the starting frame on the nipple center line under body position k. Let y be the y-component of the direction vector of the nipple centerline under body position k. For frame number, For body position k, the number of slice frames corresponding to the starting frame. Let z be the z-component of the direction vector of the nipple centerline under body position k. Let x be the x-coordinate of the stitching reference point on the i-th frame of the coronal image under the k-position. Let x be the x-coordinate of the coronal plane reference point on the nipple center line in the k-position. Let x be the x-component of the direction vector of the nipple centerline under body position k.

[0096] Based on the LLAT nipple center line Calculate the coordinates of the stitching reference point for the lateral coronal section image of frame 55. For example: .

[0097] 。

[0098] 。

[0099] 13) Calculate the overlap area between the coronal images of different positions and the median image based on the distance from the stitching reference point of each frame of coronal images in different positions to the boundary of each coronal image. and Furthermore, overlapping regions were removed from each frame of coronal section images taken from different body positions. This indicates the overlapping area between the lateral and medial views compared to the median view.

[0100] 14) Based on the calculated offset angle and Vertical offset compensation is performed on the inner and outer images after the overlapping areas have been removed. The compensation amount is calculated using the following formula.

[0101] .

[0102] .

[0103] in,( ), ( ) represent the lateral and medial views respectively. and Offset compensation in direction. The compensated coordinates are: .

[0104] .

[0105] These are the pixel coordinates of the outermost (LAT) image after offset compensation. The coordinates of the original pixel point to be compensated in the outermost (LAT) image (coordinates before compensation); These are the pixel coordinates of the inner edge (MED) image after offset compensation. The coordinates of the original pixel point to be compensated in the medial position (MED) image (coordinates before compensation).

[0106] In practical applications, the compensated image is denoted as... and The calculation process for its offset compensation is as follows: .

[0107] .

[0108] .

[0109] .

[0110] .

[0111] .

[0112] Its compensated coordinates are: .

[0113] .

[0114] 15) Finally, the compensated ultrasound images from different positions are stitched together to obtain a final, non-overlapping, smooth panoramic image. In practical applications, the compensated inner-side image will be used. Right boundary and outer side image The left boundaries are aligned to the center of the image. The left and right boundaries are used to obtain a final smooth, non-overlapping panoramic breast ultrasound image. ,like Figure 9 As shown.

[0115] This application has the following advantages: 1. Full-frame stitching reference point localization technology: By using image processing algorithms or deep learning segmentation algorithms, the nipple center coordinates of different body positions in shallow, limited frames are accurately obtained. Based on these shallow nipple center coordinates, the overall nipple centerline is fitted. The fitted centerline allows for the rapid and accurate determination of the stitching reference point coordinates for each frame of images in different body positions in a single operation.

[0116] 2. Overlapping area determination algorithm: The coordinates of the stitching reference point for each frame in different body positions are accurately obtained by using full-frame stitching reference point positioning technology. The overlapping area of ​​different body positions relative to the midline position is obtained by calculating the distance from the stitching reference point to the image boundary and then removed to ensure non-overlapping stitching of coronal section images in multiple positions.

[0117] 3. Offset compensation stitching method: Based on the offset angle of different body positions relative to the midline, offset compensation is performed on images of different body positions (except the midline) after the overlapping area has been removed to ensure the smoothness of the stitched image. The compensated image and the midline image are then stitched together to form a panoramic breast ultrasound image without uneven brightness or texture abrupt changes.

[0118] 4. By combining image processing algorithms and deep learning segmentation algorithms, a variety of options are provided for nipple localization in ABUS coronal section images. This overcomes the problem of unstable nipple localization results when using image processing algorithms alone, and can accurately extract nipple position information.

[0119] 5. By using the nipple center coordinates of shallow, finite frames in different positions to fit the nipple centerline in different positions, the stitching reference points for all frames in different positions can be accurately calculated at once, thus achieving one-time stitching of all frames in different positions. This method overcomes the problem of time-consuming processing of all frames of images based on gradient-weighted fusion stitching, improving the processing time for all images and facilitating rapid review by doctors.

[0120] 6. By calculating the distance from the stitching reference point coordinates to the image boundary, overlapping areas can be effectively identified and eliminated. Simultaneously, based on the calculated offset angle, offset compensation is applied to image data from different body positions, thereby stitching together a smooth, non-overlapping, and abruptly-free panoramic image. This method overcomes the problem of uneven brightness or abrupt texture changes in boundary areas caused by gradient-weighted stitching methods when faced with images containing significant noise and artifacts, demonstrating high applicability.

[0121] Based on the same inventive concept, this application also provides a breast ultrasound image stitching device for implementing the breast ultrasound image stitching method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations of one or more breast ultrasound image stitching device embodiments provided below can be found in the limitations of the breast ultrasound image stitching method described above, and will not be repeated here.

[0122] In one exemplary embodiment, such as Figure 12 As shown, a breast ultrasound image stitching device is provided, comprising: The selection module is used to select coronal images from breast ultrasound data of different positions of the same breast and obtain resampled images based on the coronal images.

[0123] The nipple localization module is used to locate the nipple based on the resampled image, and obtain the nipple contour and nipple center coordinates.

[0124] The mapping module is used to map the nipple contour and the nipple center coordinates to the coronal image to obtain the nipple centerline.

[0125] The offset angle calculation module is used to calculate the offset angle of different body positions relative to the midline based on the direction vector of the nipple centerline.

[0126] The stitching reference point calculation module is used to determine the stitching reference point of all coronal images in each position based on the intersection of the nipple center line and the number of frames of the coronal section of the coronal image.

[0127] The overlapping region removal module is used to calculate the overlapping region between the coronal images and the median images in different body positions based on the stitching reference point, and remove the overlapping region in the coronal images.

[0128] The longitudinal offset compensation module is used to perform longitudinal offset compensation based on the offset angle and the coronal plane image after removing the overlapping area.

[0129] The stitching module is used to stitch together the coronal images after longitudinal offset compensation to obtain a panoramic image of the breast.

[0130] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 13 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores breast ultrasound image stitching data. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a breast ultrasound image stitching method.

[0131] Those skilled in the art will understand that Figure 13 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method embodiments.

[0132] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the above-described method embodiments.

[0133] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described method embodiments.

[0134] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A breast ultrasound image stitching method, characterized by, The breast ultrasound image splicing method comprises: selecting coronal plane images in breast ultrasound data of different body positions of the same breast and obtaining resampling images based on the coronal plane images; performing nipple positioning according to the resampling images to obtain a nipple contour and nipple center coordinates; mapping the nipple contour and the nipple center coordinates to the coronal plane images to obtain a nipple center line; calculating offset angles of different body positions relative to a median position according to a direction vector of the nipple center line; determining splicing reference points of all coronal plane images in each body position according to intersections of the nipple center line and a coronal plane image coronal plane section frame number; calculating overlapping regions of coronal plane images of different body positions and the median position image according to the splicing reference points and removing the overlapping regions in the coronal plane images; performing longitudinal offset compensation according to the offset angles and the coronal plane images with the overlapping regions removed; splicing the coronal plane images after the longitudinal offset compensation to obtain a panoramic image of the breast.

2. The breast ultrasound image stitching method of claim 1, wherein, The selecting of the coronal plane images in the breast ultrasound data of different body positions of the same breast and the obtaining of the resampling images based on the coronal plane images specifically comprise: selecting coronal plane images of different depths along a coronal layer section in the breast ultrasound data of different body positions of the same breast in sequence; resampling the coronal plane images through bilinear interpolation to obtain resampling images.

3. The breast ultrasound image stitching method of claim 1, wherein, The nipple positioning according to the resampling images to obtain the nipple contour and the nipple center coordinates specifically comprises: determining a region with a nipple according to the resampling images by using an image processing algorithm or a deep learning segmentation algorithm; performing edge detection on the region with the nipple by using a Sobel edge detection operator to obtain the nipple contour; performing centroid calculation on the nipple contour to obtain the nipple center coordinates.

4. The breast ultrasound image stitching method of claim 1, wherein, The mapping of the nipple contour and the nipple center coordinates to the coronal plane images to obtain the nipple center line specifically comprises: mapping the nipple contour and the nipple center coordinates to the coronal plane images to obtain a coordinate sequence of the nipple center; performing fitting on the coordinate sequence of the nipple center by using a curve fitting method to obtain the nipple center line.

5. The breast ultrasound image stitching method of claim 1, wherein, The offset angles comprise a first offset angle and a second offset angle; a calculation formula of the offset angles is: ； ； in, This is the first offset angle. This is the second offset angle. The direction vector of the lateral nipple centerline. x Quantity, The direction vector of the median nipple centerline. x Quantity, The direction vector of the lateral nipple centerline. y Quantity, The direction vector of the median nipple centerline. y Quantity, The direction vector of the lateral nipple centerline. z Quantity, The direction vector of the median nipple centerline. z Quantity, The direction vector of the medial nipple centerline. x Quantity, The direction vector of the medial nipple centerline. y Quantity, The direction vector of the medial nipple centerline. z Quantity.

6. The breast ultrasound image stitching method of claim 1, wherein, a calculation formula of the splicing reference points is: ； wherein, is a body position index, is a lateral position, is a medial position, is a median position, is k the body position, the i coordinate of the stitching reference point on the coronal image of the y frame, is k the body position, the y coordinate of the starting coronal image on the nipple centerline, is k the body position, the y component of the nipple centerline direction vector, is a frame number, is k the body position, the section frame number corresponding to the starting frame, is k the body position, the z component of the nipple centerline direction vector, is k the body position, the i coordinate of the stitching reference point on the coronal image of the x frame, is k the body position, the x coordinate of the starting coronal image on the nipple centerline, is k the body position, the x component of the nipple centerline direction vector.

7. A breast ultrasound image stitching apparatus characterized by comprising: The breast ultrasound image splicing device comprises: a selecting module configured to select coronal plane images in breast ultrasound data of different body positions of the same breast and obtain resampling images based on the coronal plane images; a nipple positioning module configured to perform nipple positioning according to the resampling images to obtain a nipple contour and nipple center coordinates; a mapping module configured to map the nipple contour and the nipple center coordinates to the coronal plane images to obtain a nipple center line; an offset angle calculation module configured to calculate offset angles of different body positions relative to a median position according to a direction vector of the nipple center line; a splicing reference point calculation module configured to determine splicing reference points of all coronal plane images in each body position according to intersections of the nipple center line and a coronal plane image coronal plane section frame number; an overlapping area removing module configured to calculate overlapping areas of the coronal plane images of different body positions and the median plane image according to the stitching reference points and remove the overlapping areas in the coronal plane images; a longitudinal offset compensation module configured to perform longitudinal offset compensation according to the offset angle and the coronal plane images with the overlapping areas removed; a stitching module configured to stitch the coronal plane images after the longitudinal offset compensation to obtain a panoramic image of the breast.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the breast ultrasound image stitching method of any one of claims 1-6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the breast ultrasound image stitching method of any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the breast ultrasound image stitching method of any one of claims 1-6. The computer program is executed by the processor to implement the breast ultrasound image stitching method of any one of claims 1-6.

Images