A breast lesion three-dimensional reconstruction method and system

Abstract

The present application relates to a kind of breast lesion three-dimensional reconstruction methods, comprising the following steps: step S1, breast region self-rough scanning is carried out, and the depth image of breast region is acquired, and the three-dimensional model of breast and spatial position are determined;Step S2, fine scanning is carried out to lesion, and AI algorithm is used to realize the segmentation extraction of lesion feature in ultrasonic image;Step S3, with time, ultrasonic rotation angle or ultrasonic moving distance as interval, the image and position information of lesion feature during fine scanning are sequentially recorded;Step S4, the three-dimensional reconstruction of lesion is realized by three-dimensional reconstruction algorithm based on lesion boundary point cloud feature;Step S5, according to the recorded center position of lesion feature, the three-dimensional spatial position of lesion is calculated by weighted average method;Step S6, by breast center coordinates and lesion center coordinates, the relative position relationship of both based on mechanical arm coordinates is determined, and breast lesion three-dimensional reconstruction method is realized.The method can help doctor to determine the shape, size and its position in breast of lesion.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, specifically relating to a method and system for three-dimensional reconstruction of breast lesions. Background Technology

[0002] Ultrasound imaging, as a mature technology, is a non-invasive, non-radioactive, and lower-cost imaging method widely used in clinical diagnosis. Unlike the three-dimensional imaging reconstruction of traditional CT (computed tomography) and MRI (magnetic resonance imaging), three-dimensional reconstruction based on two-dimensional ultrasound images is an important research direction in the field of medical image processing. Because two-dimensional ultrasound images cannot detect the three-dimensional structure of tissues, ultrasound scans largely rely on the doctor's subjective experience. When a robot performs an autonomous ultrasound scan, without the direct involvement of a doctor, it is difficult for the doctor to intuitively determine the shape, size, and relative position of lesions to the breast, thus complicating diagnosis.

[0003] Therefore, there is an urgent need for a three-dimensional reconstruction method for breast lesions that can autonomously generate the three-dimensional structure of the breast and internal lesions without the participation of doctors in ultrasound scanning, thus helping doctors to diagnose the lesions. Summary of the Invention

[0004] To solve the above-mentioned technical problems, this invention proposes a method and system for three-dimensional reconstruction of breast lesions.

[0005] This invention relates to a method for three-dimensional reconstruction of breast lesions, comprising the following steps:

[0006] Step S1: Perform a rough autonomous scan of the breast area, record the movement trajectory and posture of the robotic arm between the two nodes of lesion appearance and lesion disappearance in the ultrasound image, obtain the depth image of the breast area, and determine the three-dimensional model and spatial position of the breast.

[0007] Step S2: Perform a detailed scan of the lesion based on the recorded robotic arm movement trajectory and posture, and use AI algorithms to segment and extract the lesion features in the ultrasound image;

[0008] Step S3: Record the images and location information of the lesion features during the detailed scanning process in an orderly manner, with time, ultrasound rotation angle or ultrasound movement distance as intervals;

[0009] Step S4: Achieve three-dimensional reconstruction of the lesion using a three-dimensional reconstruction algorithm based on the point cloud features of the lesion boundary;

[0010] Step S5: Calculate the three-dimensional spatial location of the lesion using the weighted average method based on the recorded lesion feature center location.

[0011] Step S6: Determine the relative positional relationship between the breast center coordinates and the lesion center coordinates based on the robotic arm coordinates to realize the three-dimensional reconstruction method of breast lesions.

[0012] The present invention also relates to a system for a three-dimensional reconstruction method of breast lesions, the system comprising a computer module for running the three-dimensional reconstruction method of breast lesions.

[0013] Beneficial effects

[0014] The three-dimensional reconstruction method for breast lesions of the present invention is applicable to autonomous ultrasound scanning processes without doctor intervention. By autonomously generating a three-dimensional structure of the breast and its internal lesions through scanning, the relative positional relationship between the two is determined, helping doctors to determine the shape, size, and location of the lesions within the breast. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the three-dimensional reconstruction method for breast lesions according to the present invention.

[0016] Figure 2 This is a schematic diagram of the breast positioning process of the present invention;

[0017] Figure 3 This is a schematic diagram of the application platform for the three-dimensional reconstruction method of breast lesions of the present invention;

[0018] Figure 4 This is a schematic diagram of lesion localization in this invention;

[0019] Figure 5 This is a schematic diagram of the feature image processed by point cloud extraction of lesion boundaries in this invention;

[0020] Figure 6 This is a schematic diagram of the three-dimensional reconstruction structure of the lesion in this invention;

[0021] Figure 7 This is a visual schematic diagram of the three-dimensional structure and relative positional relationship between the breast and the lesion in this invention. Detailed Implementation

[0022] The following combination Figures 1 to 7 This implementation method will be described in detail.

[0023] The three-dimensional reconstruction method for breast lesions of the present invention mainly includes the following steps:

[0024] Step S1: Perform a rough autonomous scan of the breast area, record the movement trajectory and posture of the robotic arm between the two nodes of lesion appearance and lesion disappearance in the ultrasound image, obtain the depth image of the breast area, and determine the three-dimensional model and spatial position of the breast.

[0025] The application platform of this invention is as follows Figure 3As shown, 1 is a robotic arm, and the fixing device 2 at the end of the robotic arm is used to fix the binocular camera 3 and the ultrasonic probe clamp 4. The ultrasonic probe clamp 4 is used to fix the ultrasonic probe 5.

[0026] During the coarse ultrasound scan, the movement trajectory of the robotic arm 1 is recorded between the appearance and disappearance of lesions in the ultrasound images; multi-view depth images are captured and recorded, and the recorded multi-view depth images are converted into point cloud data. The three-dimensional structure of the breast is constructed by multi-angle point cloud registration, and the spatial position of the three-dimensional breast structure is determined with the center of the bottom plane of the three-dimensional breast structure as the center point of the model.

[0027] Step S1 specifically includes:

[0028] Step S11: Use AI algorithm to detect lesions in ultrasound images and record the movement trajectory and posture of the robotic arm from the appearance of lesions in the ultrasound images until they disappear.

[0029] Step S12: Use a stereo camera to acquire and record depth images of the breast region:

[0030] The chest is placed within the visual range of the binocular camera 3. The skin around the chest is covered with a surgical drape while the chest is exposed to facilitate the extraction of chest point clouds. The breast is roughly scanned by ultrasound, and multi-view images are taken and recorded to capture depth images. The camera coordinate system is transformed into the robotic arm coordinate system through hand-eye calibration, thereby determining the position of the chest point cloud based on the robotic arm coordinate system.

[0031] Step S13: Extract point cloud data from the depth image, and optimize the breast point cloud through point cloud denoising and multi-angle point cloud registration;

[0032] The recorded multi-view depth images are converted into point cloud data, point cloud features in the chest area are extracted, and breast point cloud is constructed through multi-angle point cloud registration.

[0033] Step S14: Based on the breast point cloud arranged according to the rules, establish the three-dimensional structure of the breast through surface reconstruction;

[0034] Step S15: Calculate the position of the breast 3D model in the robot arm's base coordinate system;

[0035] Based on the points on the breast surface, a two-dimensional plane is drawn at the bottom of the model. The two point clouds A(x) with the smallest and largest x-values ​​on the x-axis in this two-dimensional plane are selected. min ,y A ,z A ) and B(x max ,y B ,z B ), where x min The minimum value on the x-axis, y A Let z be the value of point A on the y-axis. ALet x be the value of point A on the z-axis. max The maximum value on the x-axis, y B Let z be the value of point B on the y-axis. B Let B be the value on the z-axis; take the two point clouds C(x) with the smallest and largest y-values ​​on the y-axis. C ,y min ,z C ) and D(x D ,y max ,z D ), where x C Let C be the value on the x-axis, and y be the value on the x-axis. min The minimum value on the y-axis, z C Let x be the value of point C on the z-axis. D Let D be the value on the x-axis, and y be the value on the y-axis. max The maximum value on the y-axis, z D Let point D be the value on the z-axis; using these four point clouds as feature points of the bottom two-dimensional plane of the breast model, calculate the center position O of this plane. B (X,Y,Z);

[0036] in,

[0037] Center O at the bottom of the breast model B It serves as the center of the three-dimensional model of the entire breast.

[0038] Step S2: Perform a detailed scan of the lesion based on the recorded robotic arm movement trajectory and posture, and use AI algorithms to segment and extract the lesion features in the ultrasound image;

[0039] Based on the recorded robotic arm movement trajectory and posture, a certain robotic arm posture in the middle part of the trajectory is selected as the starting posture to perform a detailed scan of the lesion. The scanning method is not limited to fan-shaped scanning and rotational scanning. During the ultrasound scan, AI algorithms are used to perform image segmentation processing on the lesion in the ultrasound image and extract the lesion features.

[0040] Step S3: Record the images and location information of the lesion features during the detailed scanning process in an orderly manner, with time, ultrasound rotation angle or ultrasound movement distance as intervals;

[0041] A detailed scan of the lesion is performed, and multimodal information of the lesion during the scan is recorded at intervals of time, ultrasound rotation angle, or ultrasound movement distance, including ultrasound images, corresponding feature images, and lesion location information.

[0042] like Figure 4 As shown, during ultrasound scanning, the ultrasound probe acquires ultrasound images 8 displayed on the ultrasound device interface 6. Ultrasound images 8 can show detected lesions 9, and the actual ultrasound detection depth 7 is marked on the right.

[0043] The method for determining the location of lesions is as follows:

[0044] During the detailed scan, ultrasound images 8 on the ultrasound device interface 6 are acquired in real time using a video capture card. Based on the resolution of the acquired images, the pixel height h and width w of the ultrasound images are determined. Using the lesion center 10 in the ultrasound image as the lesion location point, the pixel coordinates o(x,y,z) of this point on the ultrasound plane are calculated. Based on the selected detection depth H and the actual width W of the ultrasound probe image, the lesion pixel coordinates are converted into actual spatial coordinates, and the spatial coordinates of the lesion on the ultrasound plane are calculated as O. N (X L ,Y L Z L );

[0045] in, Z L =0, where N represents the number of feature maps;

[0046] The lesion features in each ultrasound image are calculated based on the spatial coordinates in the ultrasound plane coordinate system, and the coordinate system of the feature center pixel is transformed to the ultrasound probe coordinate system centered on the front end of the ultrasound probe.

[0047] Step S4: Achieve three-dimensional reconstruction of the lesion using a three-dimensional reconstruction algorithm based on the point cloud features of the lesion boundary;

[0048] The 3D reconstruction process of lesions is divided into online reconstruction and offline reconstruction. Online reconstruction refers to real-time reconstruction during fine scanning, using some stored lesion boundary point cloud features, at intervals of time, ultrasound rotation angle, or ultrasound movement distance. The reconstruction interval must be greater than or equal to twice the time interval, ultrasound rotation angle, or ultrasound movement distance interval for recording lesion features. Offline reconstruction refers to 3D reconstruction using all stored lesion boundary point cloud features after a complete scan of the lesion. Online reconstruction can reconstruct the scanned lesion structure in real time during fine scanning, while offline reconstruction can only perform overall 3D reconstruction of the lesion after fine scanning is completed.

[0049] like Figure 5 As shown, the lesion feature image 12 recorded during fine scanning, and the lesion boundary point cloud features 11 were extracted;

[0050] like Figure 6 As shown, a series of regular and ordered point cloud features will be used to construct triangular patches to achieve three-dimensional reconstruction of the lesion structure.

[0051] Step S5: Calculate the three-dimensional spatial location of the lesion using the weighted average method based on the recorded lesion feature center location.

[0052] Based on the recorded series of lesion characteristics and their central locations O1, O2, O3, ..., O NThe three-dimensional spatial location of the lesion was calculated using the weighted average method. N represents the number of feature maps.

[0053] Step S6: Determine the relative positional relationship between the breast center coordinates and the lesion center coordinates based on the robotic arm coordinates to realize the three-dimensional reconstruction method of breast lesions.

[0054] Through the center coordinates of the breast O B and the coordinates of the lesion center O L The relative positional relationship between the two models was determined based on the robotic arm coordinates. Visualization software was then used to further visualize the 3D breast model and the lesion model, providing a more intuitive representation of their 3D structure and positional relationship. Figure 7 As shown, 13 is the three-dimensional structure of the breast, and 14 is the three-dimensional structure of the lesion.

[0055] The present invention also relates to a system for a three-dimensional reconstruction method of breast lesions, the system comprising a computer module for running the three-dimensional reconstruction method of breast lesions.

[0056] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the invention. They can also be reasonable combinations of the features described in the above embodiments. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for three-dimensional reconstruction of breast lesions, characterized in that, Includes the following steps: Step S1: Perform a rough autonomous scan of the breast area, record the movement trajectory and posture of the robotic arm between the two nodes of lesion appearance and lesion disappearance in the ultrasound image, obtain the depth image of the breast area, and determine the three-dimensional model and spatial position of the breast. Step S2: Perform a detailed scan of the lesion based on the recorded robotic arm movement trajectory and posture, and use AI algorithms to segment and extract the lesion features in the ultrasound image; Step S3: Record the images and location information of the lesion features during the detailed scanning process in an orderly manner, with time, ultrasound rotation angle or ultrasound movement distance as intervals; A detailed scan of the lesion was performed, and multimodal information of the lesion during the scan was recorded at intervals of time, ultrasound rotation angle, or ultrasound movement distance, including ultrasound images, corresponding feature images, and lesion location information; The method for determining the lesion location information is as follows: During detailed scanning, ultrasound images are acquired in real time from the ultrasound equipment interface using a video capture card. Based on the resolution of the acquired images, the pixel height h and width w of the ultrasound images are determined. Taking the center of the lesion in the ultrasound image as the lesion location point, the pixel coordinates of this point on the ultrasound plane are calculated. Based on the selected detection depth H and the actual imaging width W of the ultrasound probe, the pixel coordinates of the lesion are converted into actual spatial coordinates, and the spatial coordinates of the lesion in the ultrasound plane are calculated as follows: ; in, , , N represents the number of feature maps; The lesion features in each ultrasound image are calculated based on the spatial coordinates in the ultrasound plane coordinate system, and the coordinate system of the feature center pixel is transformed to the ultrasound probe coordinate system centered on the front end of the ultrasound probe. Step S4: Achieve three-dimensional reconstruction of the lesion using a three-dimensional reconstruction algorithm based on the point cloud features of the lesion boundary; The 3D reconstruction process of lesions is divided into online reconstruction and offline reconstruction. Online reconstruction refers to real-time reconstruction during fine scanning, using some stored lesion boundary point cloud features, with time, ultrasound rotation angle, or ultrasound movement distance as intervals. The reconstruction interval is required to be greater than or equal to twice the time, ultrasound rotation angle, or ultrasound movement distance interval for recording lesion features. Offline reconstruction refers to 3D reconstruction using all stored lesion boundary point cloud features after a complete scan of the lesion. Step S5: Calculate the three-dimensional spatial location of the lesion using the weighted average method based on the recorded lesion feature center location. Step S6: Determine the relative positional relationship between the breast center coordinates and the lesion center coordinates based on the robotic arm coordinates to realize the three-dimensional reconstruction method of breast lesions.

2. The method for three-dimensional reconstruction of breast lesions according to claim 1, characterized in that, Step S1 specifically includes: Step S11: Use AI algorithm to detect lesions in ultrasound images and record the movement trajectory and posture of the robotic arm from the appearance of lesions in the ultrasound images until they disappear. Step S12: Use a binocular camera to acquire and record depth images of the breast region; Step S13: Extract point cloud data from the depth image, and optimize the breast point cloud through point cloud denoising and multi-angle point cloud registration; Step S14: Based on the breast point cloud arranged according to the rules, establish the three-dimensional structure of the breast through surface reconstruction; Step S15: Calculate the position of the breast 3D model in the robot arm's base coordinate system.

3. The method for three-dimensional reconstruction of breast lesions according to claim 2, characterized in that, In step S15, a two-dimensional plane at the bottom of the model is drawn based on the point cloud on the breast surface. The two point clouds with the smallest and largest x-values ​​on the x-axis in this two-dimensional plane are then selected. and ,in The minimum value on the x-axis. Let A be the value on the y-axis. Let A be the value on the z-axis. The maximum value on the x-axis. Let B be the value on the y-axis. The z-axis value of point B; the two point clouds with the smallest and largest y-values ​​on the y-axis. and ,in Let C be the value on the x-axis. The minimum value on the y-axis. Let C be the value on the z-axis. Let D be the value on the x-axis. The maximum value on the y-axis. Let D be the value on the z-axis; using these four point clouds as feature points of the bottom two-dimensional plane of the breast model, calculate the center position of this plane. ; in, , , ; Center of the bottom of the breast model As the center of the three-dimensional model of the entire breast.

4. The method for three-dimensional reconstruction of breast lesions according to claim 1, characterized in that, In step S5, based on the recorded central locations of a series of lesion features... The three-dimensional spatial location of the lesion was calculated using the weighted average method. N represents the number of feature maps.

5. The method for three-dimensional reconstruction of breast lesions according to claim 1, characterized in that, In step S6, the coordinates of the breast center are used. and the coordinates of the lesion center The relative positional relationship between the two models was determined based on the coordinates of the robotic arm. Visualization software was then used to visualize the three-dimensional model of the breast and the three-dimensional model of the lesion, showing their three-dimensional structure and positional relationship.

6. A system for implementing the three-dimensional reconstruction method for breast lesions according to any one of claims 1 to 5, characterized in that, The system includes a computer module for running a three-dimensional reconstruction method for breast lesions.

Images