Method and system for matching tomography ultrasonic scanning result with body surface image

Abstract

The invention belongs to the technical field of medical images and ultrasonic diagnosis, and discloses a method and system for matching a tomography ultrasonic scanning result with a body surface image, and the method comprises the following steps: preparing an ultrasonic probe and an actual camera; acquiring an actually shot target area image; actual camera internal parameter information is obtained and used for setting a virtual camera; performing ultrasonic scanning and recording a timestamp to obtain three-dimensional point cloud data, and selecting a reference point; and loading the three-dimensional grid model according to a preset direction by using a virtual camera in a 3D rendering environment, matching the three-dimensional grid model with an actually shot target area image through affine transformation on the premise that a reference point is overlapped with a pre-selected space point, and displaying a matching result. The corresponding relation between the ultrasonic scanning result and the body surface image can be visually displayed, an ultrasonic technician is assisted in adjusting the scanning angle and position in real time, and the ultrasonic image quality and diagnosis accuracy are improved.

Description

Technical Field

[0001] The present invention belongs to the field of medical imaging and ultrasonic diagnosis technology, and more specifically, relates to a method and system for matching tomographic ultrasonic scanning results with body surface images, which can match the reconstructed three-dimensional ultrasonic image with the actual body surface image. Background Art

[0002] Ultrasound technology is widely used in medical imaging and has become a vital diagnostic tool due to its advantages such as being non-invasive, real-time, portable, low-cost, versatile, and high-resolution. In existing ultrasound diagnostics, doctors typically rely on experience to determine the correct probe placement and scanning angle to obtain the ideal ultrasound image. However, due to individual patient differences, the complex structure of the acquisition site, and the influence of scattering and noise on the ultrasound image itself, the actual ultrasound images obtained are often incomplete and have directional deviations, making it difficult for doctors to accurately determine whether the scan results meet expectations.

[0003] Currently, during conventional ultrasound diagnostics, doctors assess the accuracy of scans by viewing two-dimensional ultrasound images. Doctors often need to manually adjust the probe position, gradually finding the correct angle and direction through multiple scans. This operation relies on the doctor's experience and can still lead to inaccurate positioning and incomplete scanning of complex areas.

[0004] To overcome this problem, existing technologies have begun to attempt to use 3D ultrasound reconstruction to assist doctors in determining scanning direction and position. However, existing 3D ultrasound images are typically only 3D reconstructions of blood vessels or other organs, and doctors still cannot intuitively correlate the 3D reconstruction results with the surface image. Therefore, even if doctors can determine whether there is a problem with the scan by observing the 3D vascular model, it is still difficult to locate the corresponding location in the surface image and then adjust the ultrasound scanning direction.

[0005] Existing research focuses more on individual ultrasound image processing, but there is no mature solution for matching body surface images with ultrasound three-dimensional reconstruction results. Summary of the Invention

[0006] In response to the aforementioned deficiencies or improvements in the prior art, the present invention aims to provide a method and system for matching tomographic ultrasound scan results with body surface images. This method matches the tomographic ultrasound scan results with the body surface image, intuitively displaying the corresponding relationship between the ultrasound scan results and the body surface image, and assisting ultrasound technicians in real-time adjustment of the scanning angle and position (in this case, the fixture guide rail and the actual camera are treated as a single component, and the relative position or posture of the object under test and the component is adjusted; the relative position of the actual camera relative to the guide rail remains unchanged), thereby improving ultrasound image quality and diagnostic accuracy. In particular, the present invention can utilize the affine transformation matrix obtained during the initial matching as a universal affine transformation matrix. Combined with reference points selected according to a preset fixed principle, this matrix can be applied to subsequent matching of other objects under test.

[0007] To achieve the above object, according to one aspect of the present invention, a method for matching tomographic ultrasound scanning results with body surface images is provided, characterized in that it includes the following steps:

[0008] (1) Fixing the ultrasonic probe on the fixture guide rail with a clamp, so that the ultrasonic probe can move along the fixture guide rail with the drive of the motor and perform ultrasonic scanning on the target to be measured, while the starting scanning position of the ultrasonic probe on the fixture guide rail remains unchanged, and the starting scanning position is recorded as the preset starting point; fixing the actual camera outside the guide rail, and the relative position of the actual camera relative to the guide rail remains unchanged, and the actual camera can photograph the surface of the target to be measured that is being ultrasonically scanned;

[0009] (2) obtaining the target area image actually captured by the actual camera;

[0010] (3) Obtaining the internal parameter information of the actual camera and converting the parameter information into an internal parameter matrix for use in the setting of the virtual camera, so as to keep the internal parameters of the virtual camera consistent with those of the actual camera, thereby ensuring that the imaging effect of the virtual camera in the three-dimensional simulation environment is consistent with that of the actual camera;

[0011] (4) Starting the motor to move the ultrasonic probe from a preset starting point along the fixture guide rail, while performing ultrasonic scanning and recording timestamps, obtaining ultrasonic tomographic data of the target area at different probe positions at different times, and selecting reference points according to preset principles; then, generating three-dimensional point cloud data based on target detection and image segmentation algorithms, where any three-dimensional data point includes ultrasonic tomographic data and timestamp information;

[0012] (5) Converting 3D point cloud data into a 3D mesh model through a 3D reconstruction algorithm;

[0013] (6) The target area image actually shot obtained in step (2) is loaded into a 3D rendering environment as a background image, and the three-dimensional mesh model obtained in step (5) is loaded in a preset direction using the virtual camera obtained in step (3) in the 3D rendering environment, so that the three-dimensional mesh model is projected onto the background image to achieve visual simulation of the imaging process; then, under the premise that the reference point overlaps with the spatial point pre-selected in the field of view of the 3D rendering environment, the three-dimensional mesh model is processed by affine transformation so that the three-dimensional mesh model is matched with the target area image actually shot, and the matching result is displayed.

[0014] As a further preference of the present invention, the method further comprises the steps of:

[0015] (7) Save the affine transformation matrix used for the affine transformation in step (6), restore the ultrasound probe to the preset starting point, and update the target to be measured for ultrasound scanning; then, obtain the target area image actually photographed by the actual camera corresponding to the updated target to be measured, and obtain the three-dimensional mesh model and reference point corresponding to the updated target to be measured according to steps (4) and (5); then, under the perspective of the virtual camera obtained in step (3), load the target area image actually photographed corresponding to the updated target to be measured into the 3D rendering environment as a background image; then, load the three-dimensional mesh model corresponding to the updated target to be measured into the 3D rendering environment in a preset direction, and initialize it so that the reference point overlaps with the pre-selected spatial point in the field of view of the 3D rendering environment; then, use the saved affine transformation matrix to process the three-dimensional mesh model, so as to achieve direct matching between the three-dimensional mesh model and the target area image, and display the matching result.

[0016] As a further preferred embodiment of the present invention, in step (6), the affine transformation is performed manually.

[0017] Preferably, in step (6), the virtual camera is constructed using known camera parameters of an actual camera, and is consistent with the actual camera viewing angle in step (1);

[0018] The virtual camera is constructed based on the pinhole camera model, and the camera intrinsic parameter matrix is ​​in the form of The camera internal parameter information is consistent with the actual camera in step (3); wherein fx is the length of the focal length in the x-axis direction in pixels, fy is the length of the focal length in the y-axis direction in pixels, and (cx, cy) is the coordinate of the camera principal point in pixels;

[0019] Set the virtual camera extrinsic matrix to

[0020] Correspondingly, step (6) specifically comprises: placing the three-dimensional mesh model within the field of view of the virtual camera according to the initialized translation matrix, so that the reference point overlaps with the pre-selected spatial point in the field of view of the 3D rendering environment; at the same time, setting the target area image actually photographed by the actual camera obtained in step (2) as the rendering environment background; then, manually performing an affine transformation on the three-dimensional mesh model in the 3D environment until the projection shape of the three-dimensional mesh model in the field of view of the virtual camera matches the shape and position of the target to be measured in the target area image actually photographed by the actual camera obtained in step (2).

[0021] As a further preferred embodiment of the present invention, in step (4), ultrasonic scanning is performed and timestamps are recorded, specifically: ultrasonic tomographic data are continuously acquired at intervals of a preset acquisition time;

[0022] A reference point is selected according to a preset principle, specifically: a two-dimensional coordinate point in the first ultrasonic tomographic image in the ultrasonic tomographic data is selected as the reference point;

[0023] Preferably, the center point of the first ultrasonic tomographic image in the ultrasonic tomographic data is selected as the reference point, or the upper left corner vertex of the first ultrasonic tomographic image in the ultrasonic tomographic data is selected as the reference point, or the lower left corner vertex of the first ultrasonic tomographic image in the ultrasonic tomographic data is selected as the reference point, or the upper right corner vertex of the first ultrasonic tomographic image in the ultrasonic tomographic data is selected as the reference point, or the lower right corner vertex of the first ultrasonic tomographic image in the ultrasonic tomographic data is selected as the reference point.

[0024] As a further preferred embodiment of the present invention, in step (4), three-dimensional point cloud data is generated based on target detection and image segmentation algorithms, specifically: first, a region of interest (ROI) in the ultrasound image is obtained through a pre-trained target detection model, and then these ROI regions are input into a trained segmentation network model to obtain a segmented binary image, and finally, the image is converted into three-dimensional point cloud data in combination with timestamp information;

[0025] Preferably, the segmentation network model is specifically a segmentation network model constructed based on H-Dense U-Net 2D.

[0026] As a further preferred embodiment of the present invention, step (5) specifically reconstructs the discrete point cloud data into a three-dimensional mesh model based on a surface rendering algorithm.

[0027] As a further preferred embodiment of the present invention, in step (6), the affine transformation processing is one or more of rotation, translation, and scaling.

[0028] As a further preferred embodiment of the present invention, in step (3), the actual camera internal parameter information is obtained through actual measurement.

[0029] As a further preferred embodiment of the present invention, in step (1), the target to be measured is a calibration body membrane;

[0030] Correspondingly, in step (6), the affine transformation processing is achieved by performing ultrasonic acquisition, segmentation, and three-dimensional reconstruction on the calibration body membrane, and then manually aligning the coordinates after initialization.

[0031] According to another aspect of the present invention, a system for matching tomographic ultrasound scanning results with body surface images is provided, characterized in that it includes:

[0032] The probe module includes an ultrasonic probe, a fixture guide rail, and an actual camera, wherein the ultrasonic probe is fixed to the fixture guide rail by a fixture, so that the ultrasonic probe can move translationally on the fixture guide rail with the fixture driven by a motor and ultrasonically scan the target to be measured, while the starting scanning position of the ultrasonic probe on the fixture guide rail remains unchanged, and this starting scanning position is recorded as a preset starting point; the actual camera is fixed outside the guide rail, and the relative position of the actual camera relative to the guide rail remains unchanged. The actual camera can capture the surface of the target to be measured being ultrasonically scanned to obtain an image of the target area actually captured;

[0033] Virtual camera setting module: used to obtain the internal parameter information of the actual camera and convert the parameter information into an internal parameter matrix for the virtual camera setting, so as to keep the internal parameters of the virtual camera consistent with the actual camera, so that the imaging effect of the virtual camera in the 3D simulation environment is consistent with the imaging effect of the actual camera;

[0034] Ultrasonic scanning module: This module activates the motor to move the ultrasonic probe from a preset starting point along the fixture guide rail, simultaneously performing ultrasonic scanning and recording timestamps to obtain ultrasonic tomographic data of the target area at different probe positions at different times. Reference points are selected according to preset principles. Then, based on target detection and image segmentation algorithms, 3D point cloud data is generated. Any 3D data point includes ultrasonic tomographic data and timestamp information.

[0035] 3D reconstruction module: used to convert 3D point cloud data into a 3D mesh model through a 3D reconstruction algorithm;

[0036] Matching and display module: used to load the actually captured target area image as a background image into the 3D rendering environment, and use the obtained virtual camera in the 3D rendering environment to load the obtained three-dimensional mesh model in a preset direction, so that the three-dimensional mesh model is projected onto the background image to achieve visual simulation of the imaging process; then, under the premise that the reference point overlaps with the pre-selected spatial point in the field of view of the 3D rendering environment, the three-dimensional mesh model is processed through affine transformation to match the three-dimensional mesh model with the actually captured target area image, and the matching result is displayed.

[0037] As a further preference of the present invention, it also includes:

[0038] Repeated matching module: used to save the affine transformation matrix used for the affine transformation in the matching and display module, restore the ultrasound probe to a preset starting point, and update the target to be measured to be ultrasonically scanned; then, obtain the target area image actually captured by the actual camera corresponding to the updated target to be measured, and obtain the three-dimensional mesh model and reference points corresponding to the updated target to be measured according to the ultrasound scanning module and the three-dimensional reconstruction module; then, under the obtained virtual camera perspective, the actually captured target area image corresponding to the updated target to be measured is loaded into the 3D rendering environment as the background image; then, the three-dimensional mesh model corresponding to the updated target to be measured is loaded into the 3D rendering environment according to a preset direction, and initialized so that the reference point overlaps with the pre-selected spatial point in the field of view of the 3D rendering environment; thereafter, the three-dimensional mesh model is processed using the saved affine transformation matrix to achieve direct matching between the three-dimensional mesh model and the target area image, and the matching result is displayed.

[0039] Compared with the prior art, the above technical solution conceived by the present invention provides a method and system for intuitively displaying the correspondence between ultrasound scanning results and body surface images, and obtains a strategy for matching three-dimensional ultrasound images with actual body surface images. By projecting the reconstructed data onto the body surface image, real-time feedback is provided to the ultrasound technician to assist in adjusting the scanning direction and position (in this case, the fixture guide rail and the actual camera are regarded as an integral component, and the relative position or posture of the object under test and the component is adjusted; the relative position of the actual camera relative to the guide rail remains unchanged), thereby improving the quality of the ultrasound image and enhancing the accuracy of diagnosis.

[0040] The present invention can particularly utilize the affine transformation matrix obtained during the initial matching process as a universal affine transformation matrix. This matrix, combined with reference points selected according to a preset fixed principle, can be applied to subsequent matching of other objects to be measured. By fixing the virtual camera parameters and the starting scanning position of the ultrasound probe for ultrasonic tomography, the present invention achieves a unified projection viewing angle, requiring only adjustment of the affine transformation during the first matching process. After the first matching is completed, the affine transformation matrix corresponding to the corresponding affine transformation becomes a universal transformation matrix. This universal matrix can be directly applied to subsequent ultrasound reconstruction data, enabling doctors to observe the correspondence between the reconstructed three-dimensional mesh model (e.g., a reconstructed three-dimensional vascular model) and the body surface image in real time, allowing timely adjustment of the scanning strategy, thereby improving the quality of ultrasound images, the overall consistency of review, and the accuracy of diagnosis.

[0041] Based on the method of the present invention, during the first matching, for example, a calibration membrane can be used as the object to be measured. In this way, after completing the affine transformation (which can be completed manually) and obtaining the affine transformation matrix, for the subsequent n-th matching (n is greater than or equal to 2), since each matching (including the first matching) is to load the corresponding three-dimensional mesh model into the 3D rendering environment in a preset direction, and the reference points of the corresponding ultrasound tomography data are overlapped with the pre-selected spatial points in the field of view of the 3D rendering environment, the affine transformation matrix used in the subsequent n-th matching is the same as the affine transformation matrix used in the first matching. In this way, a universal affine transformation matrix is ​​calculated and directly applied to the newly obtained three-dimensional reconstructed data during the subsequent ultrasound scanning process, which can quickly achieve subsequent matching, improve matching efficiency, and ensure consistency of results. Of course, based on the method of the present invention, only the first matching can be achieved, in which case the object to be measured is the target object rather than the calibration membrane.

[0042] The present invention performs an affine transformation on the reconstructed three-dimensional mesh model (e.g., a mesh blood vessel model) in a 3D rendering environment of a virtual camera perspective to solve the problem of positioning and matching between the model and the body surface image. The core technical problem of the method for matching the results of tomographic ultrasound scanning with the body surface image is how to project the reconstructed three-dimensional ultrasound image onto the actual body surface image without additional complex equipment. Taking the blood vessels as an example, that is, projecting the reconstructed three-dimensional ultrasound image of the blood vessels onto the actual body surface image, the true extension direction and morphology of the blood vessels are intuitively displayed, thereby assisting the ultrasound technician to determine the optimal scanning angle and position and obtain a more ideal ultrasound image. The universal affine transformation matrix obtained after the matching is completed can be directly used for the ultrasound three-dimensional reconstruction data of the subsequent target, allowing the doctor to observe the correspondence between the blood vessel model and the body surface image in real time during the ultrasound scanning process, and adjust the scanning strategy (for example, assisting the ultrasound technician to adjust the probe scanning direction and position in real time, at this time, the fixture guide rail and the actual camera are regarded as an integral component, and the relative position or posture of the object being measured and the component is adjusted), thereby improving the diagnostic efficiency.

[0043] In summary, the present invention utilizes fixed camera intrinsic parameters and a unified initialization process to overlap the reference points of the ultrasound tomographic data with preselected spatial points within the field of view of the 3D rendering environment, thereby constructing a virtual rendering environment and enabling the projection of ultrasound-reconstructed point cloud data from a fixed viewing angle. In particular, the present method allows for the manual matching of the first matching pass to obtain a universal affine transformation matrix. For subsequent matching passes, this affine transformation matrix can be directly applied to different ultrasound reconstruction results, directly achieving corresponding display with the body surface image. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The present invention is a flow chart of matching the carotid artery tomographic ultrasound scanning results with the body surface image.

[0045] Figure 2 This is an effect diagram of the method of the present invention matching the point cloud data reconstructed by ultrasound with the body surface image.

[0046] Figure 3 This diagram illustrates the positional relationship between the fixture rail, the actual camera, and the inspection site used with the method of the present invention. The actual camera is fixed to the outside of the rail, and its relative position relative to the rail remains unchanged. The ultrasound probe is secured to the fixture rail with a fixture and can translate along the rail driven by a motor. The starting scanning position of the ultrasound probe on the fixture rail remains unchanged.

[0047] Figure 4 It is a flow chart of the method for matching tomographic ultrasound scanning results with body surface images in the present invention.

[0048] Figure 5 It is a schematic diagram of the data processing flow of the method for matching tomographic ultrasound scanning results with body surface images in the present invention. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0050] Based on the present invention, a system for matching tomographic ultrasound scanning results with body surface images can be constructed, including:

[0051] (1) A camera unit (i.e., an actual camera), which is used to obtain a body surface image containing the matching target area and provide camera internal parameter information; Figure 3 As shown, the actual camera is fixed outside the guide rail, and the relative position of the actual camera relative to the guide rail remains unchanged;

[0052] (2) an ultrasonic scanning unit for collecting tomographic data of the target area and transmitting the data to a data processing unit; Figure 3 As shown, the ultrasonic probe is fixed on the fixture guide rail with a clamp, so that the ultrasonic probe can move horizontally on the fixture guide rail with the clamp driven by the motor and perform ultrasonic scanning on the target to be measured. At the same time, the starting scanning position of the ultrasonic probe on the fixture guide rail remains unchanged, and the starting scanning position is recorded as the preset starting point (of course, as shown in FIG. Figure 3After the assembly system shown is completed, the ultrasound probe is detachably fixed to the fixture. When the assembly system is subsequently used to match the 1st, 2nd, ... nth tomographic ultrasound scan results with the body surface image, the relative positions of the ultrasound probe and the fixture are exactly the same during the i-th and i+1-th (i=1, ...n-1) matchings.

[0053] (3) Ultrasonic data processing unit, used to convert the collected ultrasonic image data into point cloud data of the target part, involving target detection and automatic tube wall segmentation algorithms to ensure the accuracy and reliability of the point cloud data;

[0054] (4) a point cloud data processing unit, configured to perform three-dimensional reconstruction, point cloud meshing, virtual camera projection, and matching calculation based on affine transformation to determine the corresponding relationship between the tomographic ultrasound data and the body surface image;

[0055] (5) A display unit is used to present the matching process and final matching results in a three-dimensional rendering environment to provide intuitive reference for ultrasound physicians and assist in scanning decisions.

[0056] The method for matching tomographic ultrasound scanning results with body surface images of the present invention may include the following steps:

[0057] (1) Obtain a body surface image taken by an actual camera in advance, and obtain the camera's internal and external parameters through calibration. In other words, obtain a body surface image containing the matching target entity taken by an actual camera in advance, and the camera parameters have been obtained through actual measurement.

[0058] (2) An ultrasonic scanning unit is used to obtain cross-sectional data of the target area (while selecting reference points according to a preset principle), and a corresponding point cloud model is generated using a three-dimensional reconstruction algorithm. The coordinate system of the point cloud data is fixed during reconstruction. In other words, cross-sectional data of the target area (such as the carotid artery) is obtained through an ultrasonic scanning system, and the ultrasonic data is converted into three-dimensional point cloud data using target detection and image segmentation algorithms. By using a three-dimensional reconstruction algorithm to generate a corresponding point cloud model, the reference coordinate system of the point cloud data is fixed during reconstruction.

[0059] (3) Using a surface rendering algorithm, the point cloud data is converted into a three-dimensional mesh model (e.g., a three-dimensional network model of a blood vessel; that is, a mesh blood vessel model is reconstructed from the acquired point cloud model), and matching and registration is performed in a 3D rendering environment according to the following steps:

[0060] a. Construct a virtual camera based on camera parameters that matches the actual camera's viewing angle, and set its projection model so that the two-dimensional projection generated by the virtual camera matches the viewing angle of the actual captured body surface image;

[0061] b. Place the vascular model within the virtual camera's field of view based on the initialized translation matrix (for example, the reference point of the 3D vascular mesh model can be placed at the coordinate point (0, 0, 1000) within the virtual camera's field of view, so that the reference point overlaps with this coordinate point. Alternatively, other coordinate points within the field of view can be selected, and the selected points can remain unchanged).

[0062] c. Set the image acquired by the actual camera as the rendering environment background, which is used as a matching reference;

[0063] d. manually or automatically adjust the rotation, translation, and scaling of the model so that its projected image matches the shape and position of the target object in the body surface image;

[0064] (4) After the matching is completed, all affine transformation matrices, including rotation, translation, and scaling matrices, are combined to obtain a total affine transformation matrix, which represents the overall position adjustment of the model from the initialization position to the camera's viewpoint. In other words, after the matching is completed, all affine transformation matrices in the adjustment process are combined and superimposed to obtain a total affine transformation matrix, which reflects the relative position adjustment between the model starting from the initialization position and the camera's viewpoint.

[0065] (5) Since the camera parameters, the relative position of the target object in the actual camera view and the initialization position of the model are fixed, and the reference coordinate system of the point cloud reconstruction has been pre-unified, the obtained affine transformation matrix can be applied to another object to be measured, so that the subsequent ultrasound reconstruction model can directly apply the universal matrix to achieve automatic matching between the model and the body surface image. In other words, since the camera parameters, the relative position of the entity photographed by the actual camera and the camera, and the initialization position of the model in the 3D rendering window are fixed, and the reference system of the point cloud reconstruction of different objects has been pre-unified, the obtained affine transformation matrix is ​​universal for objects of the same type. That is, for other reconstructed models, after initialization (that is, making the reference points of different batches overlap with the same spatial point pre-selected in the field of view of the 3D rendering environment), by directly applying the universal matrix, the matching and registration of the point cloud and the body surface image can be achieved.

[0066] During the first matching process, the affine transformation matrix can be manually optimized to ensure that the 3D mesh model accurately matches the morphology and position of the target area in the body surface image.

[0067] In addition, the process of transforming the model 3D point data into 2D image points can be expressed as A=P*T*I*B;

[0068] Among them, A represents the two-dimensional point of the target object in the matched image, P is the camera projection matrix, T is the total affine transformation matrix, I is the initialization matrix, and B is the three-dimensional data point.

[0069] Example 1

[0070] This embodiment provides a strategy for matching carotid artery ultrasound scan results with neck surface images, which generally includes the following steps:

[0071] Step S1: Fix the relative positions of the sampling neck, fixture guide rail and camera, then start the motion motor and ultrasound probe to collect ultrasound images and corresponding timestamps.

[0072] Step S2: inputting the collected ultrasound image into the ultrasound data processing unit to obtain the three-dimensional point cloud data of the carotid artery.

[0073] Step S3: Obtain the reconstructed three-dimensional grid model through the point cloud data processing unit and determine its corresponding relationship in the body surface image.

[0074] Wherein, step S1 is specifically as follows:

[0075] Figure 3 The diagram below shows the relationship between the inspection site (using the carotid artery as an example), the fixture guide, and the actual camera. The fixture guide is kept parallel to the neck and suspended above it, at a height that ensures a close fit between the ultrasound probe and the neck and allows for relative movement. The camera is fixed in a fixed position on the mid-perpendicular line of the guide and does not move during the entire scan. The motion motor that controls the fixture's movement is kept at a fixed starting point to ensure that the ultrasound probe maintains its initial scanning position on the fixture guide.

[0076] After completing the above operations, the motion motor and ultrasonic probe are simultaneously activated. The motor propels the fixture and ultrasonic probe along the fixed slide rail. Ultrasonic tomographic images are continuously acquired at pre-set acquisition intervals. Ultrasonic tomographic data and corresponding timestamps are collected and transmitted to the ultrasonic data processing unit. This reduces human error, ensures scanning consistency and stability, and improves scanning efficiency. The timestamp can be calculated by multiplying the acquisition interval by the acquisition sequence number (the acquisition interval is a fixed value).

[0077] Step S2 is specifically as follows:

[0078] Several collected two-dimensional carotid artery cross-sectional ultrasound images were input into the Yolo-v8 network pre-trained with 130 cases of carotid adventitial MAB as the target detection training set, and the image part containing the carotid artery was extracted as the ROI area to improve the running speed of the subsequent segmentation algorithm.

[0079] The extracted ROI is resized to 352 x 288 pixels and fed into a trained H-Dense U-Net 2D segmentation network (other segmentation networks based on H-Dense U-Net2D, such as the improved H-Dense U-Net 2D segmentation network, can also be used) to generate a segmented binary image. The MAB coordinates are then converted into 3D point data using the corresponding timestamps, resulting in a 3D point cloud of the carotid artery.

[0080] Step S3 is specifically as follows:

[0081] The camera parameters obtained through actual measurement are: lens fixed focus f, horizontal field of view angle θh, vertical field of view angle θv, image resolution is w×h pixels. This embodiment is based on the pinhole camera model, and the camera intrinsic parameter matrix can be calculated:

[0082]

[0083] in,

[0084] f is the focal length in millimeters; fx is the length of the focal length in the x-axis described in pixels; fy is the length of the focal length in the y-axis described in pixels; cx and cy are the actual positions of the principal points in pixels. These parameters can be determined by the following steps:

[0085] a. According to the pinhole camera model, assume that the following relationship exists between the lens focal length f and the sensor size s:

[0086]

[0087] Where sx and sy represent the physical dimensions of the sensor in the horizontal and vertical directions (unit: mm)

[0088] b. Assume that the image resolution is w pixels (width) and h pixels (height). The number of pixels per millimeter in the horizontal direction (i.e., the x-axis, i.e., the width of the image) and the height direction (i.e., the y-axis, i.e., the height of the image) is

[0089]

[0090] c. Convert the physical focal length f to pixel units to obtain:

[0091] fx=f*px,fy=f*py

[0092] d. It is usually assumed that the principal point (optical center) is located at the center of the image, then

[0093]

[0094] Since a reference image of the relative position of the real object and the camera has been obtained during the actual shooting process, the image fully reflects the viewing angle of the actual camera and the spatial relationship between the object and the camera. Therefore, in this solution, the setting of the camera extrinsic parameters has no substantial effect on the final matching result. Regardless of how the extrinsic parameters are set, the subsequent adjustment process through affine transformation is essentially to correct the relative position between the object and the camera, and there is duplication in the functions of the two. That is to say, since the present invention restores the relative position relationship between the actual photographed object and the actual camera by applying affine transformation to the three-dimensional mesh model to adjust its position and posture in the rendering environment, thereby achieving a viewing angle consistent with the image taken by the actual camera, it does not rely on the spatial position and posture adjustment of the virtual camera itself. Therefore, the external parameter matrix of the virtual camera does not rely on the external parameter setting of the actual camera. Therefore, in order to simplify the calculation process, the external parameter matrix can be selected:

[0095]

[0096] That is, the virtual camera is located at the world origin and has no rotation.

[0097] Based on the above discussion, the camera parameters in the embodiment are also expressed as the camera projection matrix:

[0098]

[0099] Camera parameters can also be directly measured and obtained through other methods such as camera calibration.

[0100] Point cloud data is obtained through ultrasound scanning and reconstruction, and its reference system is pre-unified during reconstruction to ensure that each 3D mesh model has a consistent coordinate system. In order to ensure that the image captured by the virtual camera and the body surface image captured by the actual camera have the same perspective, the 3D mesh model is first initialized. The specific steps are as follows:

[0101] a. Move the reference point of the 3D mesh model to a preset fixed position (e.g., [0, 0, 1000]) through a translation operation so that it appears within the field of view of the 3D rendering window (the field of view of the 3D rendering window is pre-loaded with the actual captured image of the target area as a background image, i.e., a reference image for subsequent alignment of the 3D mesh model). By doing this, 3D mesh models of different sizes and different initial positions can be translated to a unified initial observation position. For example, a fixed 2D coordinate point can be selected from the first ultrasound tomographic image as the reference point (e.g., the center point, upper left corner vertex, lower left corner vertex, upper right corner vertex, or lower right corner vertex of the 2D ultrasound tomographic image, as long as the selection principle remains fixed, such as selecting the center point). The reason for selecting the reference point according to the fixed principle is that the ultrasound scan volume is a unified 3D scan volume composed of multiple equally spaced ultrasound images. Different 3D mesh models are local structures extracted from this volume. When the ultrasound probe's initial scanning position is consistent, the resulting reference surface is fixed. Therefore, the reference point obtained from the first ultrasonic tomographic image of the ultrasonic tomographic data is a unified, objective and reusable reference point between different three-dimensional grid models.

[0102] b. Use known parameters to set the virtual camera's viewing angle to ensure that the 2D projection image captured by the virtual camera has a consistent projection model with the actual image. This step ensures that the virtual camera and the actual camera use the same internal parameters (focal length, field of view, image resolution, etc.), so that the projection result generated by the virtual camera has the same geometric characteristics as the actual captured image, which provides a basis for subsequent comparison and matching under the same viewing angle.

[0103] c. Set the pre-acquired body surface image captured by the actual camera as the rendering environment background, which serves as the reference image for subsequent matching. This can be aided by observing whether the projection of the 3D mesh model under the virtual camera aligns with the target in the background image.

[0104] Next, the initial relative position between the reconstructed three-dimensional mesh model and the actual object is adjusted. The operator performs affine transformation adjustments on the point cloud in the 3D rendering window through manual observation, including translation, rotation (i.e., rotation around the reference point), and scaling. The affine transformation matrix can be obtained through the calibration body membrane (the calibration body membrane can be obtained by the following steps: first, 3D print a rectangular shell, the geometric feature of which is that one side has a small rectangular block and a small triangular pyramid protrusion facing inward; then fill the inside with a coupling agent with different acoustic resistance and solidify it to obtain the calibration body membrane). During the manual matching process, the operator constantly observes the 2D projection results taken by the virtual camera and compares them with the pre-collected surface images until the two match in morphology, position, and direction. After the matching is completed, all local transformations are superimposed in sequence, that is, the total affine transformation matrix is ​​obtained.

[0105] like Figure 5 As shown, the entire transformation process can be expressed as: A = P*T*I*B;

[0106] Among them, A represents the two-dimensional point of the target object in the image after matching, P is the camera projection matrix, T is the total affine transformation matrix, I is the initialization matrix, and B is the three-dimensional data.

[0107] The camera projection matrix is ​​the product of the camera intrinsic parameter matrix and the camera extrinsic parameter matrix. The affine transformation matrix is ​​the sum of the products of the translation matrix, the rotation matrix, and the scaling matrix.

[0108] In the above implementation, the camera projection matrix P is a fixed parameter; the data of each reconstructed 3D mesh model has reference points selected according to the same preset principle. Furthermore, the 3D mesh model is loaded into the 3D rendering environment according to a preset orientation, and the reference points are overlapped with the same preselected spatial point within the 3D rendering environment's field of view. Therefore, the affine transformation matrix T obtained through manual adjustment is universally applicable. That is, for other ultrasound-reconstructed carotid artery models, the same P, T, and I matrices can be directly used for matching and registration, thereby accurately projecting the reconstructed 3D vascular mesh model (i.e., the simulated mesh vascular model) onto the body surface image.

[0109] The above embodiments are merely examples. For example, after the target to be measured is changed, in addition to requiring the starting scanning position of the ultrasonic probe on the fixture guide rail to remain unchanged, the ending scanning position of the ultrasonic probe can be flexibly changed according to actual conditions (of course, it cannot exceed the guide rail length).

[0110] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for matching tomographic ultrasound scanning results with body surface images, characterized in that: The following steps are involved: (1) Fixing the ultrasonic probe on the fixture guide rail with a clamp, so that the ultrasonic probe can move along the fixture guide rail with the drive of the motor and perform ultrasonic scanning on the target to be measured, while the starting scanning position of the ultrasonic probe on the fixture guide rail remains unchanged, and the starting scanning position is recorded as the preset starting point; fixing the actual camera outside the guide rail, and the relative position of the actual camera relative to the guide rail remains unchanged, and the actual camera can photograph the surface of the target to be measured that is being ultrasonically scanned; (2) obtaining the target area image actually captured by the actual camera; (3) Obtaining the internal parameter information of the actual camera and converting the parameter information into an internal parameter matrix for use in the setting of the virtual camera, so as to keep the internal parameters of the virtual camera consistent with those of the actual camera, thereby ensuring that the imaging effect of the virtual camera in the three-dimensional simulation environment is consistent with that of the actual camera; (4) Starting the motor to move the ultrasonic probe from a preset starting point along the fixture guide rail, while performing ultrasonic scanning and recording timestamps, obtaining ultrasonic tomographic data of the target area at different probe positions at different times, and selecting reference points according to preset principles; then, generating three-dimensional point cloud data based on target detection and image segmentation algorithms, where any three-dimensional data point includes ultrasonic tomographic data and timestamp information; (5) Converting 3D point cloud data into a 3D mesh model through a 3D reconstruction algorithm; (6) The target area image actually shot obtained in step (2) is loaded into a 3D rendering environment as a background image, and the three-dimensional mesh model obtained in step (5) is loaded in a preset direction using the virtual camera obtained in step (3) in the 3D rendering environment, so that the three-dimensional mesh model is projected onto the background image to achieve visual simulation of the imaging process; then, under the premise that the reference point overlaps with the spatial point pre-selected in the field of view of the 3D rendering environment, the three-dimensional mesh model is processed by affine transformation so that the three-dimensional mesh model is matched with the target area image actually shot, and the matching result is displayed.

2. The method according to claim 1, wherein: The method further comprises the steps of: (7) Save the affine transformation matrix used for the affine transformation in step (6), restore the ultrasound probe to the preset starting point, and update the target to be measured for ultrasound scanning; then, obtain the target area image actually photographed by the actual camera corresponding to the updated target to be measured, and obtain the three-dimensional mesh model and reference point corresponding to the updated target to be measured according to steps (4) and (5); then, under the perspective of the virtual camera obtained in step (3), load the target area image actually photographed corresponding to the updated target to be measured into the 3D rendering environment as a background image; then, load the three-dimensional mesh model corresponding to the updated target to be measured into the 3D rendering environment in a preset direction, and initialize it so that the reference point overlaps with the pre-selected spatial point in the field of view of the 3D rendering environment; then, use the saved affine transformation matrix to process the three-dimensional mesh model, so as to achieve direct matching between the three-dimensional mesh model and the target area image, and display the matching result.

3. The method according to claim 1, wherein: In step (6), the affine transformation is performed manually. Preferably, in step (6), the virtual camera is constructed using known camera parameters of an actual camera, and is consistent with the actual camera viewing angle in step (1); The virtual camera is constructed based on the pinhole camera model, and the camera intrinsic parameter matrix is ​​in the form of The camera internal parameter information is consistent with the actual camera in step (3); wherein fx is the length of the focal length in the x-axis direction in pixels, fy is the length of the focal length in the y-axis direction in pixels, and (cx, cy) is the coordinate of the camera principal point in pixels; Set the virtual camera extrinsic matrix to Correspondingly, step (6) specifically comprises: placing the three-dimensional mesh model within the field of view of the virtual camera according to the initialized translation matrix, so that the reference point overlaps with the pre-selected spatial point in the field of view of the 3D rendering environment; at the same time, setting the target area image actually photographed by the actual camera obtained in step (2) as the rendering environment background; then, manually performing an affine transformation on the three-dimensional mesh model in the 3D environment until the projection shape of the three-dimensional mesh model in the field of view of the virtual camera matches the shape and position of the target to be measured in the target area image actually photographed by the actual camera obtained in step (2).

4. The method according to claim 1, wherein: In step (4), ultrasonic scanning is performed and timestamps are recorded, specifically: ultrasonic tomographic data are continuously acquired at intervals of a preset acquisition time; A reference point is selected according to a preset principle, specifically: a two-dimensional coordinate point in the first ultrasonic tomographic image in the ultrasonic tomographic data is selected as the reference point; Preferably, the center point of the first ultrasonic tomographic image in the ultrasonic tomographic data is selected as the reference point, or the upper left corner vertex of the first ultrasonic tomographic image in the ultrasonic tomographic data is selected as the reference point, or the lower left corner vertex of the first ultrasonic tomographic image in the ultrasonic tomographic data is selected as the reference point, or the upper right corner vertex of the first ultrasonic tomographic image in the ultrasonic tomographic data is selected as the reference point, or the lower right corner vertex of the first ultrasonic tomographic image in the ultrasonic tomographic data is selected as the reference point.

5. The method according to claim 1, wherein: In step (4), three-dimensional point cloud data is generated based on target detection and image segmentation algorithms. Specifically, the regions of interest (ROIs) in the ultrasound image are first obtained through a pre-trained target detection model, and then these ROIs are input into a trained segmentation network model to obtain segmented binary images. Finally, the images are converted into three-dimensional point cloud data in combination with timestamp information. Preferably, the segmentation network model is specifically a segmentation network model constructed based on H-Dense U-Net 2D.

6. The method according to claim 1, wherein: Step (5) specifically reconstructs the discrete point cloud data into a three-dimensional mesh model based on a surface rendering algorithm.

7. The method according to claim 1, wherein: In step (6), the affine transformation processing is one or more of rotation, translation, and scaling.

8. The method according to claim 1, wherein: In step (3), the actual camera internal parameter information is obtained through actual measurement.

9. The method according to claim 1, wherein: In step (1), the target to be measured is a calibration body membrane; Correspondingly, in step (6), the affine transformation processing is achieved by performing ultrasonic acquisition, segmentation, and three-dimensional reconstruction on the calibration body membrane, and then manually aligning the coordinates after initialization.

10. A system for matching tomographic ultrasound scan results with body surface images, characterized in that: include: The probe module includes an ultrasonic probe, a fixture guide rail, and an actual camera, wherein the ultrasonic probe is fixed to the fixture guide rail by a fixture, so that the ultrasonic probe can move translationally on the fixture guide rail with the fixture driven by a motor and ultrasonically scan the target to be measured, while the starting scanning position of the ultrasonic probe on the fixture guide rail remains unchanged, and this starting scanning position is recorded as a preset starting point; the actual camera is fixed outside the guide rail, and the relative position of the actual camera relative to the guide rail remains unchanged. The actual camera can capture the surface of the target to be measured being ultrasonically scanned to obtain an image of the target area actually captured; Virtual camera setting module: used to obtain the internal parameter information of the actual camera and convert the parameter information into an internal parameter matrix for the virtual camera setting, so as to keep the internal parameters of the virtual camera consistent with the actual camera, so that the imaging effect of the virtual camera in the 3D simulation environment is consistent with the imaging effect of the actual camera; Ultrasonic scanning module: This module activates the motor to move the ultrasonic probe from a preset starting point along the fixture guide rail, simultaneously performing ultrasonic scanning and recording timestamps to obtain ultrasonic tomographic data of the target area at different probe positions at different times. Reference points are selected according to preset principles. Then, based on target detection and image segmentation algorithms, 3D point cloud data is generated. Any 3D data point includes ultrasonic tomographic data and timestamp information. 3D reconstruction module: used to convert 3D point cloud data into a 3D mesh model through a 3D reconstruction algorithm; A matching and display module is configured to load the actual captured image of the target area into a 3D rendering environment as a background image, and to use the obtained virtual camera in the 3D rendering environment to load the obtained three-dimensional mesh model in a preset direction, thereby projecting the three-dimensional mesh model onto the background image to achieve visual simulation of the imaging process; then, under the premise that a reference point overlaps with a pre-selected spatial point in the field of view of the 3D rendering environment, the three-dimensional mesh model is processed through affine transformation to match the three-dimensional mesh model with the actual captured image of the target area, and the matching result is displayed; Preferably, it also includes: Repeated matching module: used to save the affine transformation matrix used for the affine transformation in the matching and display module, restore the ultrasound probe to a preset starting point, and update the target to be measured to be ultrasonically scanned; then, obtain the target area image actually captured by the actual camera corresponding to the updated target to be measured, and obtain the three-dimensional mesh model and reference points corresponding to the updated target to be measured according to the ultrasound scanning module and the three-dimensional reconstruction module; then, under the obtained virtual camera perspective, the actually captured target area image corresponding to the updated target to be measured is loaded into the 3D rendering environment as the background image; then, the three-dimensional mesh model corresponding to the updated target to be measured is loaded into the 3D rendering environment according to a preset direction, and initialized so that the reference point overlaps with the pre-selected spatial point in the field of view of the 3D rendering environment; thereafter, the three-dimensional mesh model is processed using the saved affine transformation matrix to achieve direct matching between the three-dimensional mesh model and the target area image, and the matching result is displayed.

Images