A fiber duct endoscope image processing method and system

Abstract

The present application relates to the technical field of image processing, and more specifically, to a fiber duct endoscope image processing method and system, the method comprising: obtaining an original image frame and synchronized current posture parameters; corresponding the axial direction of a two-dimensional unfolded image data grid to the pullback direction of an endoscope probe, and corresponding the circumferential angle of the two-dimensional unfolded image data grid to the rotation angle of a lumen cross section; performing a reverse projection calculation to obtain a three-dimensional direction vector; obtaining a target coordinate position in the two-dimensional unfolded image data grid corresponding to the three-dimensional direction vector based on a preset standard cylindrical model; filling the color information of the pixel point to the corresponding target coordinate position; and generating a duct inner wall unfolded image based on the color information filled in the two-dimensional unfolded image data grid. The method has the advantage of being able to overcome the composite image distortion caused by both mechanical contact of the probe and liquid pressure fluctuations.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to a fiber duct endoscope image processing method and system. Background Art

[0002] To comprehensively screen the mammary ductal wall during fiberoptic ductoendoscopy, an expanded image composed of a sequence of continuous images must be generated to identify early-stage or subtle lesions. However, due to factors such as endoscopic probe motion, complex ductal structures, and operator manipulation, the original images often exhibit significant geometric distortion, compromising both stitching and diagnosis. For example, during an actual examination, the operator retracts and rotates an endoscope equipped with a micro-camera to capture images. The target duct exhibits pathological stenosis and highly compliant walls, requiring continuous perfusion of saline to expand the lumen and create an artificial lumen. During the retraction process, the probe often deforms asymmetrically due to eccentric contact with the ductal wall. The pressed side appears closer and larger, while the other side appears further away and smaller, transforming the image from a circular shape to an irregular shape. Furthermore, fluctuations in perfusion pressure induce a "breathing" effect, where the ductal wall expands and contracts. This, combined with the mechanical compression of the probe, results in dramatic changes in tissue morphology within consecutive images, severely distorting the image stitching and creating an unrealistic "peristaltic" artifact of the lesion. Therefore, how to reconstruct an inner wall expansion image without splicing artifacts that can accurately reflect the true geometric size and morphology of the tube wall lesions, thereby overcoming the composite image distortion caused by the mechanical contact of the probe and the liquid pressure fluctuation, is a challenge facing current technology.

[0003] In view of the above problems, the existing technology is in urgent need of improvement. Summary of the Invention

[0004] In order to address the deficiencies of the prior art, the present application provides a fiber duct endoscope image processing method and system, which has the advantage of being able to overcome the composite image distortion caused by the mechanical contact of the probe and the liquid pressure fluctuation, and generate an expanded image of the inner wall of the milk duct without splicing artifacts.

[0005] The present application provides a fiber duct endoscope image processing method, comprising:

[0006] Obtaining original image frames collected during the endoscope scanning process and current posture parameters synchronized with each original image frame, the current posture parameters including axial position, rotation angle, eccentric displacement and tilt angle;

[0007] Based on a preset two-dimensional expanded image data grid, the axial direction of the two-dimensional expanded image data grid corresponds to the pull-back direction of the endoscope probe, and the circumferential angle of the two-dimensional expanded image data grid corresponds to the rotation angle of the lumen cross section;

[0008] Based on the two-dimensional coordinates of each pixel in each original image frame, the current posture parameters at the corresponding moment, and the camera imaging geometry, a reverse projection calculation is performed to obtain a three-dimensional direction vector;

[0009] Based on the preset standard cylindrical model, the three-dimensional direction vector is intersected with the standard cylindrical model surface to obtain the target coordinate position in the two-dimensional unfolded image data grid corresponding to the three-dimensional direction vector;

[0010] Fill the color information of the pixel point to the corresponding target coordinate position;

[0011] The expanded image of the inner wall of the milk duct is generated according to the color information filled in the two-dimensional expanded image data grid.

[0012] Through the above scheme, the composite image distortion caused by the mechanical contact of the probe and the liquid pressure fluctuation can be overcome, the true geometric size and morphology of the duct wall lesions can be accurately reflected, and an expanded image of the inner wall of the milk duct without splicing artifacts can be generated.

[0013] To further solve the problem, the present application also proposes that filling the color information of the pixel point to the corresponding target coordinate position includes:

[0014] If multiple pixels are mapped to the same target coordinate position, obtain the color information of all pixels mapped to the same target coordinate position and the corresponding cumulative number;

[0015] The stored color information of the current target coordinate position and the color information of the newly mapped pixel point are weightedly accumulated according to the corresponding cumulative times, and the fused color information is calculated as the color information of the target coordinate position.

[0016] Through the above scheme, the color information fusion when multiple pixels are mapped to the same position can be effectively processed, thereby improving the color accuracy and smoothness of the expanded image.

[0017] To improve the solution, this application also proposes that the method further includes:

[0018] Based on the multiple historical posture parameters collected continuously, determine whether there is abnormal drift in the change trend of the axial position and rotation angle in the current posture parameters;

[0019] If abnormal drift is determined to exist, the area with significant texture features is identified as the reference area based on the color information filled in the two-dimensional expansion data grid;

[0020] Perform image registration on the reference area at the current moment and the reference area corresponding to the previous moment to obtain the corrected displacement;

[0021] The axial position and the rotation angle in the current posture parameters are corrected according to the corrected displacement to update the current posture parameters used for reverse projection calculation.

[0022] Through the above scheme, the attitude parameters can be corrected through attitude parameter drift detection and image registration, thereby improving the accuracy and stability of the unfolded image.

[0023] To improve the solution, this application also proposes to correct the axial position and rotation angle in the current posture parameters according to the correction displacement to update the current posture parameters used for reverse projection calculation, including:

[0024] Determining whether the correction displacement exceeds a preset tolerance threshold;

[0025] If the corrected displacement exceeds a preset tolerance threshold, a posture error compensation model including the influence of eccentric displacement and tilt angle is obtained;

[0026] The attitude error compensation model constructs the mapping deviation between the actual imaging path and the ideal imaging path based on the camera imaging geometry and the eccentric displacement and tilt angle in the current attitude parameters;

[0027] The axial position and rotation angle in the current posture parameters are corrected according to the mapping deviation to update the current posture parameters used for inverse projection calculation.

[0028] Through the above scheme, the mechanism of attitude parameter correction is further refined, and by introducing the attitude error compensation model, the errors caused by eccentric displacement and tilt angle are corrected more accurately.

[0029] To improve the solution, this application also proposes that the method further includes:

[0030] Based on the preset standard structure marking area, determine whether the current milk duct inner wall expansion image meets the reference structure alignment conditions;

[0031] If it is determined that the reference structure alignment conditions are not met, a prompt message for attitude parameter calibration is generated;

[0032] Acquire confirmation information in response to the posture parameter calibration prompt information, register the image of the corresponding standard structure mark area in the corresponding original image frame with the preset standard structure image, and obtain a registration offset;

[0033] The axial position and rotation angle in the current posture parameters are corrected based on the registration offset, and the current posture parameters used for inverse projection calculation are updated.

[0034] Through the above scheme, a posture parameter calibration mechanism based on standard structure marking areas is introduced to improve the global alignment accuracy of the unfolded graph.

[0035] To improve the solution, this application also proposes that the method further includes:

[0036] Based on the shape boundary features of the standard structure marking area in the unfolded image of the inner wall of the milk duct, it is determined whether there is an area missed by image acquisition;

[0037] If it is determined that there is an image acquisition omission area, generating image acquisition supplementary prompt information, the image acquisition supplementary prompt information including the axial position and rotation angle corresponding to the image acquisition omission area;

[0038] Acquire a newly acquired original image frame and its synchronized current posture parameters in response to the image acquisition supplementary prompt information;

[0039] Based on the newly acquired original image frame and the corresponding current posture parameters, the expanded image of the inner wall of the milk duct is updated.

[0040] Through the above scheme, it is possible to identify and prompt the areas missed in image acquisition, and guide supplementary acquisition to ensure the integrity of the expanded image of the inner wall of the milk duct.

[0041] To improve the solution, this application also proposes that the method further includes:

[0042] Based on the variation range of the posture parameters of each original image frame and the imaging resolution of the camera, the axial resolution and circumferential resolution of the two-dimensional unfolded image data grid are dynamically adjusted;

[0043] If, during the current acquisition process, the axial position or rotation angle change amplitude of multiple consecutive original image frames is lower than the preset grid change resolution threshold, the axial resolution and circumferential resolution of the two-dimensional unfolded image data grid are increased, and the resolution adjustment result of the two-dimensional unfolded image data grid is recorded;

[0044] The process of synchronizing the two-dimensional unfolded drawing data grid with the reverse projection calculation and the target coordinate position determination process.

[0045] Through the above scheme, the resolution of the expanded image can be dynamically adjusted according to the posture changes during the acquisition process, thereby improving the ability to present image details and data utilization efficiency.

[0046] To improve the solution, this application also proposes that the method further includes:

[0047] Based on the distribution of color information at each target coordinate position in the expanded image of the inner wall of the milk duct, multi-scale texture features are extracted and processed. The multi-scale texture features are the texture gradient, directional consistency, and local contrast of each operating area at multiple scales. The extraction and processing of multi-scale texture features includes Gaussian pyramid downsampling, Laplacian sharpening, and directional gradient filtering operations on the expanded image of the inner wall of the milk duct;

[0048] Based on multi-scale texture features, the texture salient areas in the unfolded image of the inner wall of the milk duct are identified as standard structure marking areas.

[0049] Through the above scheme, a method for automatically identifying the standard structure marking area is provided, thereby improving the automation and accuracy of the calibration process.

[0050] To improve the solution, this application also proposes that the method further includes:

[0051] Based on the color information change characteristics at multiple target coordinate positions in the expanded image of the inner wall of the milk duct, local areas with potential structural abnormality risks are identified. The color information change characteristics include the color gradient between adjacent target coordinate positions, texture boundary discontinuity, and brightness mutation amplitude;

[0052] When a local area meets the preset abnormality triggering conditions, a structural abnormality prompt information is generated, and the structural abnormality prompt information includes the axial position and circumferential angle corresponding to the local area;

[0053] Processing information in response to the structural abnormality prompt information is obtained, and the corresponding original image frame is compared with a preset standard structural image according to the processing information to determine whether there is tissue abnormality or image acquisition artifact in the local area.

[0054] Through the above solution, potential structural abnormalities can be automatically identified and a further verification mechanism can be provided to assist doctors in diagnosis.

[0055] To improve the solution, this application also proposes a fiber duct endoscope image processing system, the technical points of which are:

[0056] include:

[0057] An image acquisition module is used to obtain the original image frames collected during the endoscope scanning process and the current posture parameters synchronized with each original image frame. The current posture parameters include axial position, rotation angle, eccentric displacement and tilt angle;

[0058] a grid mapping module for, based on a preset two-dimensional expanded image data grid, making the axial direction of the two-dimensional expanded image data grid correspond to the pullback direction of the endoscope probe, and making the circumferential angle of the two-dimensional expanded image data grid correspond to the rotation angle of the lumen cross section;

[0059] A three-dimensional direction vector calculation module is used to perform reverse projection calculation based on the two-dimensional coordinates of each pixel point in each original image frame, the current posture parameters at the corresponding moment, and the camera imaging geometry to obtain the three-dimensional direction vector;

[0060] The target coordinate acquisition module is used to intersect the three-dimensional direction vector with the standard cylindrical model surface based on the preset standard cylindrical model to obtain the target coordinate position in the two-dimensional unfolded diagram data grid corresponding to the three-dimensional direction vector;

[0061] Color filling module, used to fill the color information of the pixel point to the corresponding target coordinate position;

[0062] The image generation module generates an expanded image of the inner wall of the milk duct according to the color information filled in the two-dimensional expanded image data grid.

[0063] Through the above solution, a system for implementing the above image processing method is provided, which is convenient for practical application and deployment.

[0064] In summary, the fiber duct endoscope image processing method and system provided in the present application utilizes the complete posture parameters including the eccentric displacement and tilt angle of the endoscope probe in the lumen acquired in real time, and combines the reverse projection calculation and the standard cylindrical model to accurately map the pixel points of the original image frame to the two-dimensional unfolded image data grid, thereby overcoming the composite image distortion caused by the mechanical contact of the probe and the liquid pressure fluctuation, accurately reflecting the true geometric size and morphology of the duct wall lesions, and generating a duct inner wall unfolded image without splicing artifacts. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A flowchart of a fiber duct endoscopy image processing method provided in one embodiment of the present application.

[0066] Figure 2 This is one of the flow charts of a fiber duct endoscopy image processing method provided in another embodiment of the present application.

[0067] Figure 3 This is a second flow chart of a fiber duct endoscopy image processing method provided in another embodiment of the present application.

[0068] Figure 4 This is a third flow chart of a fiber duct endoscopy image processing method provided in another embodiment of the present application.

[0069] Figure 5 This is a fourth flow chart of a fiber duct endoscopy image processing method provided in another embodiment of the present application.

[0070] Figure 6 This is a fifth flow chart of a fiber duct endoscope image processing method provided in another embodiment of the present application.

[0071] Figure 7 This is a sixth flow chart of a fiber duct endoscope image processing method provided in another embodiment of the present application.

[0072] Figure 8 This is a seventh flow chart of a fiber duct endoscopy image processing method provided in another embodiment of the present application.

[0073] Figure 9 FIG8 is a flowchart of a fiber duct endoscopy image processing method provided in another embodiment of the present application.

[0074] Figure 10 This is a flowchart of a fiber duct endoscope image processing system provided in another embodiment of the present application.

[0075] In the figure: 1. Image acquisition module; 2. Grid mapping module; 3. Three-dimensional direction vector calculation module; 4. Target coordinate acquisition module; 5. Color filling module; 6. Image generation module. DETAILED DESCRIPTION

[0076] The technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. The components of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for which protection is claimed, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0077] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined or explained in subsequent figures. Furthermore, in the description of this application, the terms "first," "second," etc. are used only to distinguish descriptions and should not be understood to indicate or imply relative importance. When performing a comprehensive screening of the inner wall of the milk duct, the traditional existing fiberoptic duct endoscopy image processing method suffers from geometric deformation of the original image sequence collected due to the movement of the endoscopic probe, the complexity of the ductal physiological environment, and the influence of the operating technique, resulting in the inability to accurately reconstruct the unfolded image of the inner wall of the milk duct.

[0078] Based on the above considerations, Figure 1 , this application proposes a fiber duct endoscope image processing method, comprising:

[0079] S1000: Acquire original image frames collected during the endoscope scanning process and current posture parameters synchronized with each original image frame, the current posture parameters including axial position, rotation angle, eccentric displacement, and tilt angle;

[0080] S2000: Based on a preset two-dimensional expanded image data grid, the axial direction of the two-dimensional expanded image data grid corresponds to the pull-back direction of the endoscopic probe, and the circumferential angle of the two-dimensional expanded image data grid corresponds to the rotation angle of the lumen cross section;

[0081] S3000: Based on the two-dimensional coordinates of each pixel point in each original image frame, the current posture parameters at the corresponding moment, and the camera imaging geometry, perform a reverse projection calculation to obtain a three-dimensional direction vector;

[0082] S4000: Based on a preset standard cylindrical model, intersect the three-dimensional direction vector with the standard cylindrical model surface to obtain a target coordinate position in the two-dimensional unfolded view data grid corresponding to the three-dimensional direction vector;

[0083] S5000: Fill the color information of the pixel point to the corresponding target coordinate position;

[0084] S6000: Generate a milk duct inner wall expansion diagram according to the color information filled in the two-dimensional expansion diagram data grid.

[0085] The current posture parameters refer to the quantitative information about the spatial position and posture of the endoscopic probe as it moves inside the milk duct, including axial position, rotation angle, eccentric displacement, and tilt angle. These can be acquired using technologies such as inertial measurement units, optical tracking systems, electromagnetic positioning systems, or image-based visual odometry. For example, these can be acquired through a sensor array integrated into the endoscopic probe or tracked by an external positioning device.

[0086] The axial direction of an endoscopic probe refers to the direction of longitudinal extension or movement of the probe, that is, the direction of the central axis extending from the rear end (grip) to the front end (lens) of the probe. For flexible or rigid endoscopes, the centerline of the optical lens assembly forms an imaginary principal axis; the axial direction is defined as the forward / backward direction of the probe as it is advanced or retracted through a space such as a milk duct, digestive tract, or lumen.

[0087] The 2D expanded image data grid is a pre-defined 2D grid structure used to store expanded image data of the milk duct inner wall. It can be represented by a data structure such as a matrix, array, or hash table. For example, a pixel matrix with M rows and N columns, where each row represents an axial position and each column represents a circumferential angle, is used.

[0088] Camera imaging geometry refers to the mathematical mapping relationship that describes how an endoscopic camera projects an object in three-dimensional space onto a two-dimensional image plane. It can be implemented using a pinhole camera model, a distortion model, or a more complex wide-angle / fisheye lens model.

[0089] Backward projection calculation involves inferring the corresponding viewing direction of each pixel in the original image frame in three-dimensional space based on the geometric relationship of the camera imaging and the current pose parameters of the endoscopic probe. This viewing direction is represented as a three-dimensional direction vector, representing a ray originating from the camera's optical center, passing through the pixel, and pointing toward the inner wall of the milk duct. This three-dimensional direction vector serves as the basis for subsequently determining the axial position and circumferential angle of the pixel in the two-dimensional unfolded image data grid. In practical implementation, a pinhole camera model can be used for backward projection calculation. For each pixel in the original image frame, its two-dimensional coordinates in the image coordinate system can be expressed as (u, v). The system pre-calibrates the endoscope camera's intrinsic parameter matrix K, which contains information such as focal length and principal point coordinates. Simultaneously, the image acquisition module provides real-time, synchronized pose parameters for the current image frame. These parameters can be converted into the camera's extrinsic parameter matrix [R|T] in the world coordinate system, where R is the rotation matrix and T is the translation vector. When performing the inverse projection calculation, the pixel point (u, v) is first converted to a point in the normalized image coordinate system, for example, by multiplying the inverse matrix of K by [u, v, 1]^T. This normalized point is then treated as a 3D point on the plane with the Z axis at 1 in the camera coordinate system, that is, [x', y', 1]^T. Finally, this 3D point is inversely transformed to the world coordinate system using the camera's extrinsic parameter matrix, resulting in a direction vector pointing from the world coordinate system origin (or the position of the camera's optical center in the world coordinate system) to the corresponding position of the pixel point in 3D space. This vector, for example, [Vx, Vy, Vz], is the obtained 3D direction vector.

[0090] The standard cylindrical model is a pre-defined cylindrical model used to approximate the geometry of the milk duct wall. It is defined using a parameterized mathematical model, for example, by specifying a radius and axis. Its purpose is to provide a unified 3D reference surface, enabling the 3D direction vectors derived from reverse projection of image pixels to intersect with a geometric surface, thereby determining the 2D coordinates of these pixels on the expanded image.

[0091] The target coordinate position is the corresponding position on the 2D expanded image data grid, determined by inversely projecting the pixels in the original image frame and intersecting them with the standard cylindrical model. This can be represented by the grid's row and column indices, for example, a (row, col) integer pair. This positions the visual information in the original image at the corresponding position in the expanded image, ensuring that the resulting expanded image reflects the geometric topology of the milk duct wall.

[0092] The core innovation of this application lies in combining the original image frames acquired during endoscopic scanning with the synchronized current posture parameters (including axial position, rotation angle, eccentric displacement, and tilt angle), and performing reverse projection calculations based on the camera imaging geometry. This allows the image pixels to be mapped to a preset standard cylindrical model surface, thereby obtaining their target coordinate positions on the 2D unfolded image data grid and ultimately generating an unfolded image of the milk duct inner wall. This method overcomes the composite image distortion caused by mechanical contact of the probe and fluid pressure fluctuations, reflects the geometric size and morphology of the duct wall lesions, and avoids the geometric deformation and stitching artifacts in traditional image stitching methods.

[0093] In some embodiments, the present application is implemented as follows: During a mammary duct examination, a small camera and an inertial measurement unit (IMU) are integrated into the front end of the endoscope probe. The IMU outputs current posture parameters in real time, including axial position, rotation angle, eccentric displacement, and tilt angle. These current posture parameters and the synchronously acquired raw image frames are transmitted to an image processing unit via wired or wireless means.

[0094] The image processing unit is pre-programmed with a 2D expanded image data grid. Its axial direction corresponds to the retraction direction of the endoscope probe, and its circumferential angle corresponds to the rotation angle of the lumen cross-section, forming a pixel matrix with adjustable resolution. For each pixel in each original image frame, the image processing unit performs a reverse projection calculation based on the pixel's 2D coordinates, the corresponding current posture parameters, and the camera imaging geometry, constructing a 3D direction vector starting from the camera's optical center and passing through that pixel.

[0095] The three-dimensional direction vector then intersects the cylindrical surface of the standard cylindrical model to obtain the three-dimensional point where it intersects the model. This three-dimensional point is mapped to a two-dimensional unfolded image data grid, its target coordinate position is determined, and the color information of the pixel point is filled into the corresponding position. If multiple pixels are mapped to the same target coordinate position, they can be fused using methods such as color weighted averaging. Finally, a ductal wall unfolded image is generated based on the filled two-dimensional unfolded image data grid, which has the ability to correct geometric distortion and is used by doctors for diagnosis. Through the above technical solution, the present application can overcome the problem of geometric deformation of the original image sequence caused by factors such as the movement of the endoscopic probe, the physiological environment of the ductal cavity, and the operation method during fiberoptic duct endoscopy. In particular, by obtaining and utilizing current posture parameters including axial position, rotation angle, eccentric displacement, and tilt angle, combined with inverse projection calculation and standard cylindrical model mapping, the present application can map pixels in the original image frame to corresponding positions in the two-dimensional unfolded image data grid, thereby solving the composite image distortion caused by mechanical contact of the probe and fluid pressure fluctuations during examination of pathologically narrowed breast duct walls.

[0096] Reference Figure 2 Another embodiment of the present application further proposes that step S5000 includes:

[0097] S5100: If multiple pixels are mapped to the same target coordinate position, obtain color information of all pixels mapped to the same target coordinate position and the corresponding cumulative number of times;

[0098] S5200: Perform weighted accumulation on the color information stored at the current target coordinate position and the color information of the newly mapped pixel point according to the corresponding cumulative times, and calculate the fused color information as the color information of the target coordinate position.

[0099] In this embodiment, the cumulative number of times refers to the frequency at which a pixel point is mapped to a specific target coordinate position or the weight of its contribution to the color information of that position. It can be determined by counting the number of times the pixel point is reversely projected to the same target coordinate position in different image frames, or by the weight assigned based on factors such as the distance between the pixel point and the target coordinate position and the imaging quality. The purpose is to quantify the contribution of each pixel point to the color information of the target coordinate position.

[0100] Weighted accumulation refers to a method of combining and calculating multiple values ​​according to their respective weights to obtain a result. This can be achieved by multiplying the color information of each pixel by its corresponding cumulative number, then adding all the products and dividing by the total cumulative number. Its purpose is to reflect the true color of the target coordinate position when fusing the color information of multiple pixels, avoiding the deviation caused by a single pixel or simple averaging.

[0101] In some embodiments, the fiber duct endoscopy image processing method proposed in this application introduces a color fusion mechanism based on cumulative times when multiple pixels are mapped to the same target coordinate position, so as to improve the color consistency and authenticity of the expanded image of the inner wall of the milk duct. Specifically, the system maintains a cumulative number for each pixel mapped to the same target coordinate position in the two-dimensional expanded image data grid, which indicates the number of times the pixel has been successfully reverse-projected to the target coordinate position during the image processing process. When a new pixel is mapped to a filled target coordinate position, the system will obtain the color information of the new pixel. The corresponding cumulative number of times , and get the color information stored at the target coordinate position and cumulative number of times , which is the historical information that has been integrated at this location. The system uses the following weighted average formula to update the color information:

[0102]

[0103] in: : The fused color information after the current target coordinate position is updated; : The color information stored at the current target coordinate position; : Color information of the newly mapped pixel; : The cumulative number of times the target coordinate position has been accumulated; : The cumulative number of newly mapped pixels. This calculation process can be continuously iterated. Whenever a new pixel is mapped to the target coordinate position, the update and Through the weighted fusion mechanism described above, the contribution of each pixel to the color information of the target coordinate position is proportional to its cumulative number of times, thus avoiding jumps or artifacts caused by inconsistent color information in overlapping image areas and improving the visual coherence and diagnostic reliability of the expanded image of the inner wall of the milk duct.

[0104] With this technical solution, when multiple pixels are mapped to the same target coordinate location in the 2D expanded image data grid, color information is no longer simply randomly selected or averaged. Instead, the color information of all mapped pixels and their corresponding cumulative counts are obtained and weightedly added together to calculate fused color information. This processing method avoids color distortion and ensures that the color information at the target coordinate location truly reflects the actual situation. Ultimately, this improves the quality of the expanded image of the inner wall of the milk duct, providing a visual basis for subsequent diagnosis.

[0105] Reference Figure 3 Another embodiment of the present application further proposes that the method further includes:

[0106] S7000: Based on the historically collected continuous historical posture parameters, determine whether there is abnormal drift in the change trend of the axial position and rotation angle in the current posture parameters;

[0107] S8000: If it is determined that abnormal drift exists, based on the color information filled in the two-dimensional expanded image data grid, an area with significant texture features is identified as a reference area;

[0108] S9000: performing image registration on the reference region at the current moment and the reference region corresponding to the previous moment to obtain a corrected displacement;

[0109] S10000: Correct the axial position and rotation angle in the current posture parameters according to the corrected displacement to update the current posture parameters used for reverse projection calculation.

[0110] Among them, in this embodiment, judging whether there is abnormal drift in the changing trend of the axial position and the rotation angle in the current posture parameters means evaluating whether it deviates from the expected or stable motion pattern by analyzing the motion trajectory of the endoscopic probe in the axial and rotational directions. Specifically, it can be achieved by establishing a mathematical model or statistical model of the change of the posture parameters over time, such as based on Kalman filtering, sliding average or trend prediction algorithm, to predict the expected value or range of change of the current posture parameters. The axial position and rotation angle of the current posture parameters actually collected are compared with the expected value or range. If the deviation exceeds the preset deviation threshold, it is determined that there is abnormal drift.

[0111] Identifying regions with significant texture features as reference areas involves selecting those with visually unique, easily distinguishable texture patterns within the two-dimensional unfolded image data grid. Regions with significant texture features can be areas with rich detail, high contrast, or unique geometric shapes within the image, such as duct branching points, vascular intersections, lesion edges, or glandular openings. These regions can provide sufficient information during image registration, thereby improving the accuracy and robustness of the match. Identifying these regions can be achieved by analyzing the local gradient, edge density, corner distribution, or using specific texture analysis algorithms.

[0112] Image registration refers to the process of spatially aligning two or more images through an algorithm. Here, the reference area image identified at the current moment is compared and matched with the reference area image corresponding to the previous moment to determine the relative geometric transformation between them, including translation, rotation or scaling. This can be achieved through feature point matching, region correlation matching or optimization-based methods. Its purpose is to quantify the manifestation of abnormal drift of posture parameters on the image, so as to obtain an accurate corrected displacement. Therefore, the corrected displacement refers to the actual spatial offset of the reference area at the current moment relative to the reference area at the previous moment obtained by image registration. It directly reflects the degree of drift of the endoscopic probe in axial position and rotation angle. The corrected displacement can be a two-dimensional vector containing axial and circumferential pixel-level or spatial coordinate offsets.

[0113] Correcting the axial position and rotation angle in the current posture parameters means applying the obtained corrected displacement in reverse to the current posture parameters. For example, if the image registration result shows that the image has undergone a certain displacement in the axial direction, the displacement is converted into the corresponding adjustment amount of the endoscope axial position, and added to or subtracted from the current axial position parameter. Similarly, for the correction of the rotation angle, the rotation amount of the image can be converted into the rotation angle adjustment amount of the endoscope. Its purpose is to correct the inaccurate posture parameters caused by abnormal drift, and ensure that the subsequent reverse projection calculation is based on more accurate posture information. The solution of the present application effectively improves the accuracy of the unfolded image of the inner wall of the milk duct by introducing a detection and correction mechanism for abnormal drift of posture parameters.

[0114] In some preferred embodiments, the present application is implemented as follows: To improve the geometric accuracy of expanded images of the inner wall of the milk duct, this application proposes a posture parameter calibration mechanism based on abnormal drift detection and image registration correction. First, a time-varying dynamic model is constructed for the axial position and rotation angle of the current posture parameters. A Kalman filter is then used to predict the state based on multiple historically collected posture parameters. The Kalman filter outputs the predicted value and covariance of the current posture parameter. If the deviation between the actual collected value and the predicted value exceeds a preset statistical threshold, the current posture parameter is considered to have abnormal drift. Once abnormal drift is detected, the system identifies regions with significant texture features from the color information filled in the 2D expanded image data grid, which serve as reference regions. The reference regions can be obtained using image feature detection algorithms, such as Canny edge detection, Harris corner extraction, or SIFT / SURF feature point extraction. These feature points or local image patches have texture stability, facilitating cross-time image comparison. The system then performs image registration between the current reference region and the reference region at the previous time. By extracting and matching SIFT feature points in the reference areas at two moments, a geometric transformation model is constructed by combining descriptor similarity with the RANSAC algorithm to extract the corrected displacement in the axial direction and rotation angle. For example, if the alignment shows an offset of Δx pixels in the axial direction of the unfolded image, it can be converted into an axial correction ΔL of the endoscope probe in physical space based on the axial resolution of the unfolded image; if there is a rotational offset Δθ, it is converted into a rotation angle correction. Finally, ΔL and Δθ are used to correct the axial position and rotation angle in the current posture parameters, respectively, to obtain the updated current posture parameters for subsequent inverse projection calculations. This method ensures that the posture input is consistent with the actual endoscope position, improving the spatial accuracy and diagnostic value of the unfolded image.

[0115] Reference Figure 4 Another embodiment of the present application further proposes that step S10000 includes:

[0116] S10100: Determine whether the corrected displacement exceeds a preset tolerance threshold;

[0117] S10200: If the corrected displacement exceeds a preset tolerance threshold, a posture error compensation model including the influence of eccentric displacement and tilt angle is obtained;

[0118] S10300: The attitude error compensation model constructs a mapping deviation between the actual imaging path and the ideal imaging path based on the camera imaging geometry and the eccentric displacement and tilt angle in the current attitude parameters;

[0119] S10400: Correct the axial position and rotation angle in the current posture parameters according to the mapping deviation to update the current posture parameters used for reverse projection calculation.

[0120] Among them, in this embodiment, the preset tolerance threshold refers to the maximum acceptable error range allowed for the correction displacement during the posture parameter correction process. It can be implemented using empirical values, statistical analysis results, or fixed values ​​set based on system accuracy requirements. Its purpose is to avoid unnecessary corrections to small, insignificant posture deviations, thereby preventing excessive corrections from introducing new errors or increasing the computational burden.

[0121] The posture error compensation model refers to a mathematical model used to quantify and correct the influence of the eccentric displacement and tilt angle of the endoscope probe in the lumen on the imaging path. It includes the influence of eccentric displacement and tilt angle. It can be implemented by using a geometric model based on physical optics principles, a regression model trained by machine learning methods, or a lookup table method. Its purpose is to accurately describe the deviation between the actual imaging path and the ideal imaging path, and provide a basis for subsequent posture parameter correction.

[0122] Method for establishing attitude error compensation model:

[0123] 1. Build a geometric model based on the principles of physical optics, typically involving a mathematical description of the imaging process of an endoscope camera. This involves establishing an accurate camera model, such as a pinhole camera model, and determining its internal parameters (such as focal length, principal point coordinates, and distortion coefficients). Furthermore, the ideal geometry of the duct must be defined, such as a standard cylinder, with its axes aligned with a world coordinate system. When the endoscopic probe undergoes eccentric displacement (e.g., the probe center deviates from the duct's central axis) or tilt (e.g., the probe axis is no longer parallel to the duct's central axis) within the duct, its pose in the world coordinate system changes. The geometric model describes this pose change using rotation and translation matrices and calculates how points on the duct wall are transformed from the world coordinate system to the current probe camera coordinate system. These points are then projected onto the image plane according to the camera model. By comparing the projected positions in the ideal pose (no eccentric displacement or tilt) with the projected positions in the actual pose, the mapping deviation between the actual and ideal imaging paths can be constructed. For example, a mathematical function can be established that takes the eccentric displacement and the tilt angle as input and outputs the offset of the pixel point on the image plane.

[0124] 2. Training a regression model using machine learning methods requires a large dataset to learn the complex relationship between decentering and tilt angles and imaging deviations. Constructing this dataset typically involves conducting experiments in a controlled environment, such as using a milk duct model (i.e., a phantom) with known geometry and landmarks. During the experiment, the decentering and tilt angles of the endoscopic probe are precisely measured within the phantom using a high-precision external positioning system (such as an optical tracking system). Simultaneously, the endoscopic camera captures images. By analyzing the difference between the actual and ideal projection positions of known landmarks in the image, the corresponding mapping deviations are obtained. These measured decentering, tilt angles, and mapping deviations constitute the training samples. A regression model, such as a multilayer perceptron (a type of neural network), can be trained to learn the mapping relationship from input (decentering and tilt angles) to output (mapping deviations). During training, the model adjusts its internal parameters to minimize the error between the predicted and actual deviations. For example, a neural network can be constructed whose input layer receives the X and Y components of decentering and the pitch and yaw components of tilt angles, and whose output layer outputs the U and V offsets of pixels on the image plane.

[0125] 3. To implement the attitude error compensation model using the lookup table method, a lookup table containing discrete attitude parameters and corresponding mapping deviations must be pre-built. This lookup table can be generated through extensive simulations or dense sampling in controlled experiments. During the construction process, the possible range of eccentricity and tilt angle values ​​is discretized to form a multidimensional grid. For each discrete point in the grid, that is, for each specific combination of eccentricity and tilt angle, the corresponding imaging mapping deviation is precisely calculated or measured. These deviation values ​​are stored in a table. In practice, when the current eccentricity and tilt angle are obtained, the system searches the lookup table for the closest discrete point and directly reads the corresponding mapping deviation. If the actual attitude parameters do not correspond to the discrete points in the table, interpolation methods (such as linear or bilinear interpolation) can be used to estimate the mapping deviation. For example, a two-dimensional lookup table can be constructed, where one-dimensional indices represent discrete eccentricity values ​​(such as 0 mm, 0.5 mm, 1.0 mm), and the other-dimensional indices represent discrete tilt angle values ​​(such as 0 degrees, 5 degrees, 10 degrees). Each cell in the table stores the pixel offset (ΔU, ΔV) of the image center point on the image plane for that combination of eccentric displacement and tilt angle. When the system detects a probe eccentric displacement of 0.7 mm and a tilt angle of 7 degrees, it looks up the corresponding deviation values ​​of 0.5 mm and 1.0 mm, 5 degrees, and 10 degrees in the table, and then interpolates to calculate the precise deviation corresponding to 0.7 mm and 7 degrees.

[0126] Mapping deviation refers to the geometric difference between the actual imaging path and the ideal imaging path due to factors such as the eccentric displacement and tilt angle of the endoscope probe. It can be achieved by pixel coordinate offset, angular deviation or displacement vector in three-dimensional space. Its purpose is to quantify the impact of these errors on image projection in order to perform accurate posture parameter correction.

[0127] In some preferred embodiments, the present application is implemented as follows: After the system obtains the corrected displacement through image registration, it first compares it with a preset tolerance threshold. For example, the threshold for axial displacement can be set to 0.5 mm, and the threshold for rotation angle can be set to 2 degrees. If the corrected displacement exceeds this threshold in any dimension, such as an axial displacement of 0.8 mm, further error compensation is considered necessary. At this point, the system activates or loads a pre-established posture error compensation model. This model can be a lookup table calibrated based on the endoscope probe structure, camera parameters, and lumen characteristics, or a neural network model trained using extensive experimental data. The model internally stores the offset of the camera imaging center relative to the ideal position and image distortion parameters for different combinations of eccentric displacement and tilt angle. This posture error compensation model is then used, combined with the real-time eccentric displacement and tilt angle data obtained from the current posture parameters (e.g., an eccentric displacement of 1.2 mm and a tilt angle of 3 degrees), as well as the inherent imaging geometry of the endoscope camera, to calculate the mapping deviation between the actual imaging path and the ideal imaging path. This mapping deviation can manifest as a two-dimensional offset vector of pixels on the image plane or a slight deflection angle of the light direction in three-dimensional space. For example, the model may calculate that due to decentration and tilt, the image center is offset by 5 pixels to the lower right and the image as a whole is rotated by 0.1 degrees. Finally, based on the calculated mapping deviation, the axial position and rotation angle in the current pose parameters are corrected. For example, if the mapping deviation indicates that the image center is offset to the lower right, this may indicate that the probe has incompletely corrected axial and rotational deviations. The system then applies these deviations to the current axial position and rotation angle, for example, by fine-tuning the axial position by 0.05 mm and the rotation angle by 0.02 degrees to offset the effects of decentration and tilt. After this correction, the updated pose parameters are used in the subsequent backprojection calculation, ensuring greater geometric accuracy in the generated ductal wall unfolding. Ultimately, the resulting ductal wall unfolding more faithfully reflects the geometry of the duct wall and the actual size of the lesion, avoiding image distortion and artifacts caused by pose deviation.

[0128] Reference Figure 5 Another embodiment of the present application further proposes that the method further includes:

[0129] S11000: Based on a preset standard structure marking area, determining whether the current expanded image of the inner wall of the milk duct meets the reference structure alignment condition;

[0130] S12000: If it is determined that the reference structure alignment condition is not met, a posture parameter calibration prompt message is generated;

[0131] S13000: Acquire confirmation information in response to the posture parameter calibration prompt information, register the image of the corresponding standard structure mark area in the corresponding original image frame with the preset standard structure image, and obtain a registration offset;

[0132] S14000: Correct the axial position and rotation angle in the current posture parameters based on the registration offset, and update the current posture parameters used for reverse projection calculation.

[0133] Among them, the preset standard structure marking area in this embodiment refers to an area on the inner wall of the milk duct with specific identification attributes, which can be achieved by using the milk duct opening, branch point, or artificially implanted markers, and its purpose is to provide a reference benchmark for the alignment of the milk duct inner wall expansion map.

[0134] The reference structure alignment condition refers to the criterion for judging whether the standard structure marked area in the current milk duct inner wall expansion diagram is consistent with the expected position and shape. It can be achieved by image feature matching, geometric deviation threshold or manual visual inspection results. Its purpose is to evaluate the alignment quality of the expansion diagram.

[0135] The posture parameter calibration prompt information refers to the calibration instruction issued by the system to the operator when it detects that the alignment conditions are not met. It can be implemented by screen pop-up windows, voice prompts or flashing indicator lights. Its purpose is to guide the operator to correct the posture parameters; confirmation information refers to the operator's response to the calibration prompt, which can be implemented by the user clicking the confirmation button, voice commands or specific gesture recognition. Its purpose is to start the subsequent calibration process.

[0136] The preset standard structure image refers to an image of a standard structure marking area that is pre-stored and serves as an alignment reference. It can be implemented using a standard anatomical atlas, historical normal case images, or manually drawn template images. Its purpose is to provide an accurate reference template for image registration; the registration offset refers to the calculated position or angle deviation between the standard structure marking area in the current image and the preset standard structure image. It can be implemented using a translation vector, rotation angle, or affine transformation parameters. Its purpose is to quantify the degree of deviation of the posture parameters.

[0137] In some preferred embodiments, the present application is implemented as follows: After generating an expanded image of the milk duct inner wall, the system can automatically or upon user triggering determine whether the expanded image satisfies the reference structure alignment criteria. The reference structure marking areas can be anatomical structures such as milk duct branch points and nipple openings. The system pre-stores target position and shape templates for these structures in the ideal expanded image.

[0138] The system uses image feature extraction methods (such as SIFT or ORB) to identify structural landmarks in the current unfolded image. It then matches feature points against a pre-set template and calculates the matching error using a geometric transformation model. If this error exceeds a set tolerance threshold, the system will indicate alignment deviation in the current unfolded image and generate a pose parameter calibration prompt, guiding the user to confirm the calibration process through the user interface or audio effects.

[0139] After receiving user confirmation, the system retrieves the original image frame corresponding to the current expanded image and extracts an image segment containing the standard structure marker area. This image segment is then registered with the preset standard structure image. The registration algorithm can use cross-correlation, phase correlation, or RANSAC-based feature point matching, outputting the translation and rotation between the images as the registration offset.

[0140] The system corrects the current posture parameters based on the registration offset. For example, if the registration result shows that the image is offset by 2 mm in the axial direction and 5 degrees in the clockwise direction, the axial position in the current posture parameters will be reduced by 2 mm and the rotation angle will be reduced by 5 degrees. The corrected current posture parameters will be used to update the reverse projection calculation process, thereby improving the spatial alignment accuracy of the milk duct inner wall expansion diagram to the standard structure. By introducing the alignment condition judgment, calibration prompts, image registration and posture parameter correction mechanism based on the standard structure marking area, it can be ensured that the reference structure in the generated milk duct inner wall expansion diagram always maintains an accurate alignment state. This improves the geometric accuracy of the milk duct inner wall expansion diagram, thereby providing a reliable basis for doctors to judge the structure of the milk duct inner wall and avoiding misjudgment or missed diagnosis due to image alignment deviation.

[0141] Reference Figure 6 Another embodiment of the present application further proposes that the method further includes:

[0142] S15000: Based on the shape boundary features of the standard structure marking area in the expanded image of the inner wall of the milk duct, determine whether there is an image acquisition missed area;

[0143] S16000: If it is determined that there is an image acquisition omission area, generate image acquisition supplementary prompt information, the image acquisition supplementary prompt information including the axial position and rotation angle corresponding to the image acquisition omission area;

[0144] S17000: Acquire a newly acquired original image frame and its synchronized current posture parameters in response to the image acquisition supplementary prompt information;

[0145] S18000: Based on the newly acquired original image frame and the corresponding current posture parameters, update the milk duct inner wall expansion map.

[0146] Among them, in this embodiment, the shape boundary characteristics of the standard structure marking area refer to the quantifiable attributes such as the geometric contour, edge clarity, continuity and deviation from the preset standard shape presented by the standard structure marking area on the unfolded diagram of the inner wall of the milk duct, which can be achieved by using edge detection algorithms, contour analysis methods or feature extraction networks based on deep learning.

[0147] The missed area in image acquisition refers to the area on the expanded image of the inner wall of the milk duct that should have been imaged and filled with color information by the endoscope. However, due to the probe movement trajectory, field of view obstruction or lumen deformation during the actual acquisition process, this area is not effectively covered, resulting in the appearance of missing or incomplete blank areas on the expanded image. It can be manifested as discontinuous boundaries, distorted shapes or partial missing areas of the standard structure marking area on the expanded image. Its purpose is to clarify the target range that needs to supplement the image acquisition.

[0148] Image acquisition supplementary prompt information refers to the data instructions automatically generated by the system and provided to the operator based on the detected image acquisition omission area, which is used to guide the re-acquisition of images. It can include the axial position of the omission area in the milk duct, the circumferential rotation angle range, and the possible required probe eccentric displacement or tilt angle suggestion. It can be presented to the operator in the form of text, graphic overlay or voice command.

[0149] The solution of the present application further introduces a mechanism for detecting and supplementing the integrity of the expanded image based on generating the expanded image of the inner wall of the milk duct and calibrating the posture parameters.

[0150] In some preferred embodiments, the present application is implemented as follows. After generating an expanded image of the inner wall of the duct and completing posture parameter calibration, the system can activate an integrity detection module. This module can perform image analysis on the standard structural marker area in the expanded image of the inner wall of the duct. For example, the Canny edge detection algorithm can be used to extract the boundary of the area, and then its geometric shape can be identified using a Hough transform or contour tracking algorithm. If the detected shape boundary deviates significantly from the preset standard structural shape, such as a break, gap, or irregular depression, the system can determine that an area has been missed in the image acquisition.

[0151] If a missed area is detected, the system can calculate its corresponding axial position and rotation angle based on the coordinate range of the missed area on the expanded image. For example, if the missed area is located within a specific row and column range on the expanded image, the system can convert it into a specific distance in the pullback direction of the endoscopic probe and an angular range in the circumferential direction. The system can then generate supplementary image acquisition prompt information, which can highlight the missed area on the display in the form of a graphical overlay and prompt the operator to move the endoscopic probe to the corresponding axial position and rotation angle for supplementary acquisition via text or voice.

[0152] The operator re-operates the endoscopic probe according to the prompts, and the system can obtain newly acquired original image frames and their synchronized current posture parameters in real time. These newly acquired image frames can contain complete visual information of previously omitted areas. Finally, the system can use these newly acquired original image frames and the corresponding current posture parameters to perform the reverse projection calculation again, filling the corresponding blank areas of the milk duct inner wall expansion image with the pixel information of the new image. If the newly acquired image overlaps with the original image, the system can use weighted averaging or pixel fusion algorithms to ensure that the updated expansion image is smooth and seamless, thereby generating a complete and continuous expansion image of the milk duct inner wall.

[0153] Reference Figure 7 Another embodiment of the present application further proposes that the method further includes:

[0154] S19000: Dynamically adjusts the axial and circumferential resolutions of the 2D unfolded image data grid based on the range of attitude parameter variations of each original image frame and the camera imaging resolution.

[0155] S20000: If, during the current acquisition process, the axial position or rotation angle change amplitude of multiple consecutive original image frames is lower than a preset grid change resolution threshold, the axial resolution and circumferential resolution of the two-dimensional unfolded image data grid are increased, and the resolution adjustment result of the two-dimensional unfolded image data grid is recorded;

[0156] S21000: Synchronizing the two-dimensional unfolded image data grid with the reverse projection calculation process and the target coordinate position determination process.

[0157] Among them, the range of change of posture parameters refers to the variation of the axial position and rotation angle of the endoscope probe in a continuous time period when the endoscope probe moves in the milk duct. It can be obtained by differential or statistical analysis of the continuously collected posture parameters. Its purpose is to reflect the speed and direction of the probe movement; the camera imaging resolution refers to the pixel density of the image captured by the endoscope camera, that is, the number of pixels contained in a unit area. It can be determined by the physical parameters of the camera hardware itself. Its purpose is to characterize the fineness of the image details; dynamic adjustment refers to the automatic change of system parameters or behaviors according to real-time or quasi-real-time data input. It can be achieved by using preset rules, lookup tables or adaptive algorithms. Its purpose is to enable the system to adapt to changing environmental conditions; axial resolution and circumferential resolution refer to the sampling density of the two-dimensional unfolded image data grid in the axial direction (corresponding to the probe pull-back direction) and the circumferential direction (corresponding to the rotation angle of the lumen section). It can be adjusted by adjusting the grid unit. The size or number of the elements is changed, and its purpose is to control the degree of detail presented in the unfolded image; the grid change resolution threshold refers to a preset value used to judge whether the movement of the endoscope probe is slow enough or the change in the rotation angle is small enough, thereby triggering the grid resolution improvement. It can be determined by empirical values, statistical analysis results or user-set values. Its purpose is to avoid unnecessary frequent resolution adjustments; reverse projection calculation refers to mapping the pixel points in the original image frame from the two-dimensional image plane to the lumen surface in the three-dimensional space. It can be achieved by coordinate transformation based on the camera's internal parameters, external parameters and geometric model. Its purpose is to determine the position of the pixel points in the three-dimensional space; the process of determining the target coordinate position refers to mapping the pixel point position in the three-dimensional space to the specific coordinate point on the two-dimensional unfolded image data grid after the reverse projection calculation. It can be obtained by intersecting the three-dimensional direction vector with the standard cylindrical model surface. Its purpose is to flatten the three-dimensional information onto the two-dimensional unfolded image.

[0158] In some preferred embodiments, the present application is specifically implemented as follows: During the endoscopic scanning process, the image acquisition module continuously acquires raw image frames and synchronized posture parameters. A posture analysis unit can calculate the axial position change amplitude (for example, by comparing the axial position difference between adjacent frames) and the rotation angle change amplitude (for example, by comparing the rotation angle difference between adjacent frames) between consecutive raw image frames in real time. At the same time, the system pre-stores the optical parameters of the endoscope camera, including its imaging resolution. A grid resolution adjustment module can dynamically adjust the axial and circumferential resolution of the two-dimensional unfolded image data grid based on these real-time calculated posture parameter change ranges and the preset camera imaging resolution.

[0159] In fiberoptic ductoscopic image processing, the rules for dynamically adjusting the grid resolution of the 2D unfolded image data are determined based on the changes in the endoscopic probe's posture parameters during scanning and the camera's inherent imaging capabilities. Specifically, the grid resolution adjustment module monitors and calculates the range of posture parameter changes for each raw image frame in real time. This primarily includes the axial position change and rotational angle change of the endoscopic probe. The system also considers the preset camera imaging resolution, which is the inherent pixel density of the camera hardware and determines the level of image detail it can capture. Dynamic resolution adjustment is triggered when the endoscopic probe moves slowly or rotates finely within the duct. For example, if the posture analysis unit detects that the axial position change for multiple consecutive raw image frames consistently falls below a preset grid change resolution threshold and / or the rotational angle change consistently falls below another preset grid change resolution threshold, the system determines that the image is currently in an area requiring greater detail. Once these conditions are met, i.e., probe motion becomes smooth or fine, the grid resolution adjustment module increases the axial and circumferential resolutions of the 2D unfolded image data grid. This improvement means that the number of pixels per unit length or unit angle in the corresponding area of ​​the unfolded image increases, enabling more detailed visualization of the duct wall's texture and structural details. For example, if the system detects that the axial position change within five consecutive original image frames is less than 0.1 mm and the rotation angle change is less than 0.5 degrees, the grid resolution adjustment module can decide to increase the axial resolution of the 2D unfolded image grid from 100 pixels per millimeter to 200 pixels per millimeter, while simultaneously increasing the circumferential resolution from 10 pixels per degree to 20 pixels per degree. This dynamic adjustment aims to optimize the quality and information content of the unfolded image of the milk duct wall. In areas with slow probe motion, the increased resolution captures more detail, facilitating the detection of subtle lesions. In areas with faster probe motion, the lower resolution can be maintained to avoid oversampling and unnecessary computational burden, while ensuring image integrity. The adjusted resolution is recorded and applied simultaneously to the subsequent backprojection calculation and target coordinate determination processes to ensure consistency and accuracy throughout the image reconstruction process. For example, if the posture analysis unit detects that the axial position change of five consecutive raw image frames is less than 0.1 mm and the rotation angle change is less than 0.5 degrees, indicating that the probe is moving slowly or rotating finely, the grid resolution adjustment module can decide to increase the axial resolution of the 2D unfolded image data grid from 100 pixels per millimeter to 200 pixels per millimeter and the circumferential resolution from 10 pixels per degree to 20 pixels per degree. The parameters of this resolution increase, such as the increased resolution value and the triggering conditions, are recorded in a resolution log file.Subsequently, when the three-dimensional direction vector calculation module performs the reverse projection calculation, and when the target coordinate acquisition module determines the target coordinate position, the currently effective grid resolution is read from the resolution log file and the calculation is based on this resolution. For example, if the current grid resolution is increased, the reverse projection calculation will map the pixels to a denser grid point, and the target coordinate position will also be determined on a finer grid, thereby ensuring that the color information finally filled in the expanded image of the inner wall of the milk duct has higher detail accuracy. This dynamic adjustment and synchronization mechanism can automatically improve the local detail expression of the expanded image when the endoscopic probe is carefully observed or slowly pulled back, while maintaining high efficiency during rapid scanning and avoiding unnecessary computational burden.

[0160] Reference Figure 8 Another embodiment of the present application further proposes that the step of identifying the standard structure mark area includes:

[0161] S22000: Extracting and processing multi-scale texture features based on the distribution of color information at each target coordinate position in the expanded image of the inner wall of the milk duct. The multi-scale texture features are texture gradients, directional consistency, and local contrast of each operating area at multiple scales. The extraction and processing of multi-scale texture features includes performing Gaussian pyramid downsampling, Laplacian sharpening, and directional gradient filtering on the expanded image of the inner wall of the milk duct.

[0162] S23000: Based on multi-scale texture features, it identifies texture-significant areas in the expanded image of the inner wall of the milk duct and uses them as standard structural marker areas.

[0163] Among them, multi-scale texture features refer to the collection of visual attributes such as repeatability, directionality, roughness, etc. of pixel grayscale or color distribution in an image at different spatial resolutions or observation scales. They can be extracted using a variety of mathematical models and algorithms, such as wavelet transform, Gabor filter bank, or local binary pattern (LBP).

[0164] Texture gradient refers to the spatial rate of change of pixel grayscale or color in an image, which reflects the intensity and direction of the texture. It can be calculated using gradient operators such as the Sobel operator, Prewitt operator, or Roberts operator.

[0165] Directional consistency refers to the consistency or dominant direction of texture within a local area. This can be analyzed using methods such as the structure tensor, Hessian matrix, or Fourier transform, aiming to describe the direction and arrangement of textures. Local contrast refers to the difference between pixel grayscale or color values ​​within a local area of ​​an image, reflecting the clarity and saliency of the texture. It can be calculated using statistics such as the local mean, standard deviation, or entropy. Gaussian pyramid downsampling involves sequentially applying a Gaussian smoothing filter and downsampling the image to generate a series of image levels of varying resolution. This can be achieved by convolving the image with a Gaussian kernel function and then sampling every other row or column. Laplacian sharpening emphasizes edges and details by enhancing high-frequency components of an image. This can be achieved by convolving the image with the Laplacian operator. Directional gradient filtering uses a set of direction-sensitive filters to extract gradient information in different directions. This can be achieved using Gabor filters, directional Sobel filters, or Log-Gabor filters. Texture salient regions refer to areas in an image that have unique or prominent texture patterns. These areas are visually easy to perceive and distinguish, and can be identified using methods based on local contrast, texture entropy, or texture energy. Standard structure marker regions refer to specific areas in the unfolded image of the inner wall of the milk duct that have stable, recognizable morphological or textural features. These areas can be used as reference benchmarks for image registration or posture calibration, and may include milk duct branch points, glandular openings, or specific vascular textures. The solution of the present application automatically identifies standard structure marker areas with important reference value by conducting an in-depth analysis of the color information distribution of each target coordinate position in the unfolded image of the inner wall of the milk duct.

[0166] In some preferred embodiments, the present application is implemented as follows: After obtaining an expanded image of the inner wall of the milk duct, the image can be preprocessed, such as by noise removal and brightness normalization, to ensure the accuracy of color information. To extract multi-scale texture features, the following steps can be employed: First, a Gaussian pyramid downsampling operation is performed on the expanded image of the inner wall of the milk duct to generate a series of image levels with different resolutions. For example, a 3- or 5-layer pyramid structure can be generated, with each layer having half the resolution of the previous layer. Next, a Laplacian sharpening operation is performed on each layer of the pyramid to enhance image edges and detail information. Then, a directional gradient filtering operation is performed on each sharpened layer of the image. For example, a Gabor filter bank can be used, which includes multiple filters with different orientations and frequencies to capture texture gradient information in multiple directions, such as 0 degrees, 45 degrees, 90 degrees, and 135 degrees. Through these operations, texture gradients, directional consistency, and local contrast can be calculated for each operation area at multiple scales. For example, texture gradients can be obtained by calculating the horizontal and vertical gradient magnitudes of an image. Directional consistency can be determined by analyzing the statistical distribution of local gradient directions, such as by calculating the variance or entropy of the gradient directions within a local region. Local contrast can be obtained by calculating the standard deviation or the difference between the maximum and minimum pixel grayscale values ​​within a local region. To identify texture-salient regions, a texture saliency map can be constructed based on the multi-scale texture features extracted above. For example, a weighted sum of features such as texture gradient, directional consistency, and local contrast can be considered, or these features can be classified using a machine learning model (such as a support vector machine or neural network) to distinguish between texture-salient and non-salient regions. The texture saliency map is then thresholded and segmented, marking regions with a saliency score above a preset threshold as texture-salient regions. These identified texture-salient regions, such as the junction of milk duct branches, the edges of glandular openings, or vascular textures with specific morphologies, can be used as standard structural markers for subsequent posture parameter calibration.

[0167] Reference Figure 9 Another embodiment of the present application further proposes, further comprising:

[0168] S24000: Identifies local areas with potential structural abnormalities based on color variation characteristics at multiple target coordinate locations in an expanded image of the inner wall of the milk duct. Color variation characteristics include color gradients between adjacent target coordinate locations, texture boundary discontinuities, and brightness mutation amplitudes.

[0169] S25000: When a local area meets a preset abnormality trigger condition, a structural abnormality prompt information is generated. The structural abnormality prompt information includes the axial position and circumferential angle corresponding to the local area;

[0170] S26000: Acquire processing information in response to the structural abnormality prompt information, and compare the corresponding original image frame with a preset standard structural image according to the processing information to determine whether there is tissue abnormality or image acquisition artifact in the local area.

[0171] Color information variation features refer to spatial differences or discontinuities in the color attributes of pixels within an image. Specifically, these can include color gradients, texture boundary discontinuities, and brightness jump amplitudes. Their purpose is to capture visual cues that may indicate abnormalities in the image. The color gradient between adjacent target coordinates refers to the rate of change in color values ​​between two spatially adjacent target coordinates in an unfolded image of the inner wall of a milk duct. This can be quantified by calculating the difference or norm of each component in RGB or HSV color space, and its purpose is to reflect the severity of color transitions. Texture boundary discontinuities refer to the phenomenon in which the texture pattern or structure in an image suddenly interrupts or significantly changes within a certain region. These can be identified using edge detection algorithms or local texture analysis methods, and their purpose is to indicate abnormal tissue structures or lesion boundaries. Brightness jump amplitude refers to the degree to which the brightness value of a local region in an image changes dramatically relative to the surrounding area. This can be quantified using local contrast calculations or brightness histogram analysis, and its purpose is to reveal potential illumination anomalies, reflectance anomalies, or brightness variations within the tissue itself. Preset abnormality trigger conditions refer to a set of rules or thresholds used to determine whether a local area has a potential risk of structural abnormality. These can be set based on clinical experience, pathological features, or machine learning model training results. Their purpose is to provide objective criteria for identifying abnormal areas. Structural abnormality prompt information refers to the warning data generated and provided to the user by the system when a local area with a potential risk of structural abnormality is identified. This information can include the axial position and circumferential angle of the local area on an expanded image of the milk duct inner wall. Its purpose is to guide the physician to quickly locate and focus on the suspicious area. Processing information refers to the user or system's response or subsequent operation instructions to the structural abnormality prompt information. This can include viewing the original image frame, zooming in on the area, adjusting image display parameters, or requesting further diagnosis. Its purpose is to provide physicians with flexible options for handling abnormal areas. Preset standard structural images refer to reference images representing normal or typical tissue structures used for comparison with the original image frames of the suspicious local area. These images can be sourced from a database of healthy milk duct images or expertly annotated representative images. Their purpose is to assist physicians in distinguishing tissue abnormalities from image acquisition artifacts.

[0172] This approach is combined with previous methods based on texture features to identify standard structural marker regions, forming a more comprehensive and robust image analysis system.

[0173] In some preferred embodiments, the present application is implemented as follows: After generating an expanded image of the inner wall of the milk duct, the system can traverse each target coordinate position in the expanded image and calculate the color information variation characteristics between it and adjacent target coordinate positions. For example, for color gradients, the Sobel operator or Prewitt operator can be used to calculate the gradients of the R, G, and B color channels of the expanded image separately. The gradient values ​​of each channel are then combined to obtain a comprehensive color gradient. For texture boundary discontinuities, the Canny edge detection algorithm can be used to identify edges in the image and analyze their continuity or intensity. Alternatively, texture descriptors such as local binary patterns (LBP) can be used to quantify local texture uniformity. When the LBP value shows a significant jump within a local region, a texture boundary discontinuity can be considered. For brightness jump amplitude, the standard deviation or the difference between the maximum and minimum values ​​of pixel brightness within the local region can be calculated. When this difference exceeds a preset threshold, a brightness jump is considered. The system sets a color gradient threshold, triggering an abnormality alert when the color gradient of a local area exceeds this threshold; a texture discontinuity index, also triggering an abnormality alert when it exceeds a certain value; or a brightness mutation threshold, also triggering an alert when the local brightness changes dramatically. These thresholds are trained and optimized based on extensive clinical data to ensure their sensitivity and specificity for real lesions. Once a local area meets any one or more of these abnormality trigger conditions, the system immediately generates a structural abnormality alert. This alert can be displayed in text form on the user interface, such as "Suspicious area detected: axial position 15.2mm, circumferential angle 90 degrees," and the local area can be highlighted or marked on an expanded image of the milk duct wall. Upon receiving this alert, the physician can click on the marked area, and the system will then retrieve processing information, such as automatically retrieving the original image frame sequence corresponding to the local area and displaying it in an enlarged form on the screen. The physician can then compare this original image frame with a pre-set standard structural image. The preset standard structural image can be an atlas containing various normal tissue morphologies, or a normal tissue feature template trained using a machine learning model. Through this comparison, doctors can determine whether the abnormality in a certain area is caused by a true tissue lesion (such as a polyp or tumor) or artifacts during the image acquisition process (such as brief contact between the probe and the tube wall or excessive or insufficient local illumination), thereby helping doctors make more accurate diagnostic decisions.

[0174] Reference Figure 10 , the present application further proposes a fiber duct endoscope image processing system, comprising:

[0175] Image acquisition module 1, used to obtain the original image frames collected during the endoscope scanning process and the current posture parameters synchronized with each original image frame, the current posture parameters including axial position, rotation angle, eccentric displacement and tilt angle;

[0176] A grid mapping module 2 is configured to, based on a preset two-dimensional expanded image data grid, correspond the axial direction of the two-dimensional expanded image data grid to the pullback direction of the endoscope probe, and correspond the circumferential angle of the two-dimensional expanded image data grid to the rotation angle of the lumen cross section;

[0177] The three-dimensional direction vector calculation module 3 is used to perform reverse projection calculation based on the two-dimensional coordinates of each pixel point in each original image frame, the current posture parameters at the corresponding moment, and the camera imaging geometric relationship to obtain the three-dimensional direction vector;

[0178] The target coordinate acquisition module 4 is used to intersect the three-dimensional direction vector with the standard cylindrical model surface based on the preset standard cylindrical model to obtain the target coordinate position in the two-dimensional unfolded diagram data grid corresponding to the three-dimensional direction vector;

[0179] Color filling module 5, used to fill the color information of the pixel point to the corresponding target coordinate position;

[0180] The image generation module 6 generates an expanded image of the inner wall of the milk duct according to the color information filled in the two-dimensional expanded image data grid.

[0181] The solution of the present application divides the fiber duct endoscopy image processing process into multiple independent modules with clear functions, thereby constructing a system with a clear structure and clear collaborative relationship.

[0182] In some preferred embodiments, the present application is implemented as follows: The image acquisition module 1 can consist of a miniature CMOS image sensor integrated into the front end of the endoscope probe, combined with a six-degree-of-freedom inertial measurement unit as a posture sensor, transmitting image data and posture data in real time to a main control unit via an SPI or I2C interface. The grid mapping module 2 can be a two-dimensional array pre-allocated in the main control unit's memory, such as a floating-point array, where rows represent axial positions, columns represent circumferential angles, and each cell stores color information. The three-dimensional direction vector calculation module 3 can run on the main control unit's digital signal processor and, using pre-calibrated camera intrinsic and extrinsic parameter matrices, perform a reverse projection transformation on each pixel in each frame, converting pixel coordinates into three-dimensional direction vectors relative to the probe coordinate system. The target coordinate acquisition module 4 can utilize a standard cylinder represented by a mathematical model, such as a cylindrical surface with a fixed radius, to calculate the intersection of the three-dimensional direction vector and the cylindrical surface and convert the intersection coordinates into axial and circumferential indices in the two-dimensional unfolded image grid. The color filling module 5 writes the RGB color values ​​of the corresponding pixels in the original image directly to the target coordinates calculated by the raster mapping module. If multiple pixels are mapped to the same location, they can be processed by overwriting the average value or the most recent value. The image generation module 6 can be a graphics rendering engine. It reads the color data filled in the raster mapping module and renders it into a bitmap image file that can be output to the display, ultimately presenting it on the doctor's diagnosis screen.

[0183] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A fiber duct endoscope image processing method, characterized in that: include: Acquire original image frames collected during the endoscope scanning process and current posture parameters synchronized with each of the original image frames, wherein the current posture parameters include axial position, rotation angle, eccentric displacement and tilt angle; Based on a preset two-dimensional expanded image data grid, the axial direction of the two-dimensional expanded image data grid corresponds to the pull-back direction of the endoscopic probe, and the circumferential angle of the two-dimensional expanded image data grid corresponds to the rotation angle of the lumen cross section; Based on the two-dimensional coordinates of each pixel point in each of the original image frames, the current posture parameters at the corresponding moment, and the camera imaging geometric relationship, a reverse projection calculation is performed to obtain a three-dimensional direction vector; Based on a preset standard cylindrical model, the three-dimensional direction vector is intersected with the standard cylindrical model surface to obtain a target coordinate position in the two-dimensional unfolded view data grid corresponding to the three-dimensional direction vector; Fill the color information of the pixel point to the corresponding target coordinate position; A milk duct inner wall expansion diagram is generated according to the color information filled in the two-dimensional expansion diagram data grid.

2. The fiber duct endoscopy image processing method according to claim 1, characterized in that: Filling the color information of the pixel point to the corresponding target coordinate position includes: If multiple pixel points are mapped to the same target coordinate position, obtaining color information of all pixel points mapped to the same target coordinate position and the corresponding cumulative number of times; The color information stored at the current target coordinate position and the color information of the newly mapped pixel point are weightedly added according to the corresponding cumulative times, and the fused color information is calculated to be the color information of the target coordinate position.

3. The fiber duct endoscopy image processing method according to claim 1 or 2, characterized in that: The method further comprises: Based on a plurality of historical posture parameters collected continuously, determining whether there is abnormal drift in the change trend of the axial position and the rotation angle in the current posture parameters; If it is determined that abnormal drift exists, then based on the color information filled in the two-dimensional expanded image data grid, an area with significant texture features is identified as a reference area; Performing image registration on the reference area at the current moment and the reference area corresponding to the previous moment to obtain a corrected displacement; The axial position and the rotation angle in the current posture parameters are corrected according to the corrected displacement to update the current posture parameters used for reverse projection calculation.

4. The fiber duct endoscopy image processing method according to claim 3, characterized in that: The step of correcting the axial position and the rotation angle in the current posture parameters according to the corrected displacement to update the current posture parameters used for reverse projection calculation includes: Determining whether the corrected displacement exceeds a preset tolerance threshold; If the corrected displacement exceeds the preset tolerance threshold, obtaining a posture error compensation model including the influence of eccentric displacement and tilt angle; The posture error compensation model constructs a mapping deviation between the actual imaging path and the ideal imaging path based on the camera imaging geometric relationship and the eccentric displacement and tilt angle in the current posture parameters; The axial position and the rotation angle in the current posture parameters are corrected according to the mapping deviation to update the current posture parameters used for reverse projection calculation.

5. The fiber duct endoscopy image processing method according to claim 1, characterized in that: The method further comprises: Based on a preset standard structure marking area, determining whether the current expanded image of the inner wall of the laticifer meets a reference structure alignment condition; If it is determined that the reference structure alignment conditions are not met, a prompt message for attitude parameter calibration is generated; Acquiring confirmation information in response to the posture parameter calibration prompt information, registering an image of a corresponding standard structure mark area in the corresponding original image frame with a preset standard structure image, and acquiring a registration offset; The axial position and the rotation angle in the current posture parameters are corrected based on the registration offset, and the current posture parameters used for reverse projection calculation are updated.

6. The fiber duct endoscopy image processing method according to claim 5, characterized in that: The method further comprises: Based on the shape boundary features of the standard structure marking area in the expanded image of the inner wall of the milk duct, it is determined whether there is an image acquisition omission area; If it is determined that there is an image acquisition omission area, generating image acquisition supplementary prompt information, the image acquisition supplementary prompt information including the axial position and rotation angle corresponding to the image acquisition omission area; Acquire the newly acquired original image frame and the synchronized current posture parameter thereof in response to the image acquisition supplementary prompt information; The milk duct inner wall expansion image is updated based on the newly acquired original image frame and the corresponding current posture parameter.

7. The fiber duct endoscopy image processing method according to claim 6, characterized in that: The method further comprises: Dynamically adjusting the axial resolution and circumferential resolution of the two-dimensional unfolded image data grid based on the range of change of the posture parameters of each of the original image frames and the imaging resolution of the camera; If, during the current acquisition process, the amplitude of the change in the axial position or rotation angle of multiple consecutive frames of the original image frames is lower than a preset grid change resolution threshold, the axial resolution and the circumferential resolution of the two-dimensional expanded image data grid are increased, and the resolution adjustment result of the two-dimensional expanded image data grid is recorded; The process of synchronizing the two-dimensional unfolded image data grid with the reverse projection calculation process and the process of determining the target coordinate position.

8. The fiber duct endoscopy image processing method according to claim 5, characterized in that: The method further comprises: Extracting and processing multi-scale texture features based on the distribution of color information at each target coordinate position in the expanded image of the inner wall of the milk duct, wherein the multi-scale texture features are texture gradients, directional consistency, and local contrast of each operation area at multiple scales. The extraction and processing of the multi-scale texture features includes performing Gaussian pyramid downsampling, Laplacian sharpening, and directional gradient filtering operations on the expanded image of the inner wall of the milk duct; Based on the multi-scale texture features, a texture-significant region in the expanded image of the inner wall of the milk duct is identified as the standard structure marker region.

9. The fiber duct endoscopy image processing method according to claim 8, characterized in that: The method further comprises: Based on the color information change characteristics at multiple target coordinate positions in the expanded image of the inner wall of the milk duct, local areas with potential structural abnormality risks are identified, wherein the color information change characteristics include the color gradient, texture boundary discontinuity, and brightness mutation amplitude between adjacent target coordinate positions; When the local area meets the preset abnormality trigger condition, generating structural abnormality prompt information, the structural abnormality prompt information includes the axial position and circumferential angle corresponding to the local area; Processing information in response to the structural abnormality prompt information is acquired, and the corresponding original image frame is compared with a preset standard structural image according to the processing information to determine whether there is tissue abnormality or image acquisition artifact in the local area.

10. A fiber duct endoscope image processing system, characterized in that: include: An image acquisition module, configured to acquire original image frames acquired during the endoscope scanning process and current posture parameters synchronized with each of the original image frames, wherein the current posture parameters include axial position, rotation angle, eccentric displacement, and tilt angle; a grid mapping module for, based on a preset two-dimensional expanded image data grid, making the axial direction of the two-dimensional expanded image data grid correspond to the pullback direction of the endoscope probe, and making the circumferential angle of the two-dimensional expanded image data grid correspond to the rotation angle of the lumen cross section; a three-dimensional direction vector calculation module, configured to perform a reverse projection calculation based on the two-dimensional coordinates of each pixel point in each of the original image frames, the current posture parameters at the corresponding moment, and the camera imaging geometric relationship to obtain a three-dimensional direction vector; a target coordinate acquisition module, configured to intersect the three-dimensional direction vector with the surface of the standard cylindrical model based on a preset standard cylindrical model, and obtain a target coordinate position in the two-dimensional unfolded view data grid corresponding to the three-dimensional direction vector; A color filling module, used to fill the color information of the pixel point to the corresponding target coordinate position; The image generation module generates an expanded image of the inner wall of the milk duct according to the color information filled in the two-dimensional expanded image data grid.

Images